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ABSTRACT 


Candidate  models  for  predicting  the  reli¬ 
ability  growth  that  could  be  expected  to  occur  during 
the  development  of  an  electronic  system  are  evaluated. 
Development  programs  which  conducted  reliability 
growth  testing  are  reviewed.  The  reliability  data 
generated  during  these  tests  are  analyzed  relative 
to  the  candidate  models.  A  procedure  for  predicting 
reliability  growth  based  on  equipment  characteristics 
and  program  attributes  is  described,  along  with  guide¬ 
lines  for  conducting  reliability  growth  tests. 
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EXECUTIVE  SUMMARY 


The  Reliability  Growth  Prediction  Study  was  conducted 
to  develop  a  methodology  for  predicting  the  probable  reliabil¬ 
ity  growth  characteristics  for  avionics  electronic  equipment 
based  on  specific  equipment  characteristics  and  development 
program  attributes.  A  secondary  objective  was  to  provide  guide¬ 
lines  for  the  selection  of  reliability  growth  program  elements. 

A  data  base  of  historical  programs  which  had  conducted 
reliability  growth  testing  was  assembled.  Nine  different  avi¬ 
onic  systems  and  30  equipment  items  (Line  Replaceable  Units) 
comprised  the  data  base.  The  data  base  included  characteriza¬ 
tions  of  the  development  programs  and  equipment  items,  as  well 
as  failure/time  histories  for  the  equipments  as  observed  during 
growth  testing. 

Several  reliability  growth  models  were  investigated 
to  provide  a  framework  for  reliability  growth  prediction. 
Three  models  were  selected  for  analysis  of  the  historical  data: 
the  Duane  model;  the  Amy  Material  Systems  Analysis  Activity 
(AMSAA)  model;  and  the  IBM  model.  Data  analyses  included  sta¬ 
tistical  tests  of  the  goodness-of- f it  of  the  models  to  the 
data  and  development  of  relationships  between  growth  model 
parameters  and  equipment  characteristics/program  attributes. 

Both  the  Duane  and  AMSAA  models  were  found  to  yield 
reasonably  good  fits  to  most  of  the  data  sets.  However,  both 
were  found  to  have  limited  utility  as  predictive  tools  because 
of  the  empirical  nature  of  the  model  parameters.  Character¬ 
istics  of  the  failure-time  history  data  tended  to  mask  out  any 


underlying  relationships  between  the  parameters  and  equipment 
characteristics  and/or  program  attributes. 

The  IBM  model  was  found  to  provide  a  more  workable 
methodology  for  reliability  growth  prediction  because  its 
parameters  lend  themselves  more  easily  to  an  engineering  inter¬ 
pretation.  Based  on  these  interpretations  and  analyses  on  a 
limited  number  of  data  sets,  a  reliability  growth  prediction 
procedure  was  developed.  The  procedure  allows  prediction  of 
expected  reliability  growth  based  on  information  that  is  avail¬ 
able  at  the  start  of  a  development  program,  namely,  equipment 
complexity  and  maturity,  equipment  operating  hours  to  be  accumu 
lated  during  testing,  and  the  test  environmental  profile  rela¬ 
tive  to  the  operational  profile. 

An  important  conclusion  is  that  reliability  growth 
prediction  and  planning  should  focus  more  on  classification  of 
failure  modes  and  corrective  action  identification,  'rather 
than  on  the  slope  of  a  line  fitted  to  cumulative  failure- time 
data  plotted  on  a  log-log  scale.  It  is  also  recommended  that 
verification  of  corrective  action  effectiveness  be  a  major 
element  of  reliability  growth  planning. 
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1. 


INTRODUCTION 


1 . 1  BACKGROUND 

There  has  been  an  increased  emphasis  in  recent  years 
on  the  use  of  reliability  growth  testing  in  Full  Scale  Devel¬ 
opment  programs  for  electronic  systems.  This  emphasis  has  re¬ 
sulted  from  the  recognition  that  one  of  the  most  cost-effective 
ways  to  achieve  high  operational  reliability  is  to  mature  a 
system  design  before  committing  to  production  and  deployment 
of  the  system.  Reliability  growth  testing  subjects  prototype 
systems  to  a  prolonged  period  of  operation  in  an  environment 
designed  to  surface  failure  modes  and,  hence,  permit  the  identi¬ 
fication  of  corrective  actions,  such  as  design  changes  or  im¬ 
proved  production  processes,  for  incorporation  into  production 
systems.  In  most  cases,  the  growth  test  includes  an  environ¬ 
mental  profile  (e.g. ,  temperature,  vibration,  on-off  cycling) 
which  is  an  acceleration  of  the  mission  environment,  the  pur¬ 
pose  being  to  stimulate  equipment  failures  within  the  limited 
time  available  for  testing. 

A  fundamental  question  relating  to  reliability  growth 
testing  is  "What  is  the  degree  of  reliability  growth  achieved 
during  such  a  test?"  Considerable  research  has  been  performed 
addressing  this  question,  the  research  generally  being  within 
the  following  two  categories: 

•  Formulation  of  mathematical  models  which 
are  purported  to  represent  the  reliabil¬ 
ity  growth  process 

•  Case  studies  of  reliability  growth  pro¬ 
grams  conducted  for  specific  systems. 


These  two  areas  have  come  together  in  case  studies  in  which 
specific  growth  models  have  been  applied  to  data  generated 
during  a  reliability  growth  test  program.  The  two  most  com¬ 
monly  applied  models  in  such  studies  are: 


•  The  Duane  Model 

•  The  Army  Material  Systems  Analysis  Activ¬ 
ity  (AMSAA)  Model. 
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Both  of  these  models  are  consistent  with  the  frequently  observed 
phenomenon  that  measured  cumulative  failure  rate  versus  cumula¬ 
tive  operating  time  closely  plots  as  a  straight  line  on  log-log 
paper.  The  principal  difference  is  that  the  Duane  model  con¬ 
siders  reliability  growth  to  be  deterministic  in  nature,  whereas 
the  AMSAA  model  is  probabilistic  and  also  recognizes  the  possi¬ 
bility  that  a  reliability  growth  program  may  be  conducted  in  a 
series  of  stages.  Both  models  have  become  institutionalized 
to  an  extent  in  MIL-STD-1635  and  MIL-HDBK-189 ,  respectively. 

Both  the  Duane  and  AMSAA  models  are  parametric,  a 
principal  parameter . being  reliability  growth  rate,  and  the 
model  parameters  must  be  estimated  to  apply  the  model  in  a 
particular  reliability  growth  program.  To  date,  the  model 
parameter  estimation  process  has  been  accomplished  by  fitting 
the  model  to  reliability  growth  data,  using  either  statistical 
or  "eyeball"  techniques,  and,  in  effect,  letting  the  fit  deter¬ 
mine  the  values  of  the  parameters.  This  is  entirely  satisfac¬ 
tory  for  measuring  reliability  growth  during  the  progress  of 
the  test,  or  for  predicting  subsequent  reliability  growth  once 
a  sufficient  amount  of  data  have  been  generated  to  support  an 
adequate  fit.  However,  this  estimation  method  is  of  limited 
value  in  predicting  reliability  growth  before  testing  has 
started.  Clearly,  such  a  prediction  capability  is  desirable 
in  order  to  determine  whether  reliability  growth  testing  should 


£ 
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be  conducted  for  a  specific  system  development  program,  i.e., 
if  it  is  cost-effective,  and,  if  so,  the  extent  of  testing 
that  is  necessary  to  achieve  program  objectives.  All  that 
really  is  known  today  are  the  boundaries  of  the  parameters 
(e.g.  ,  a  =  0.1  to  0.7  for  the  Duane  model  reliability  growth 
rate),  that  the  reliability  growth  rate  is  somehow  related  to 
the  "aggressiveness"  of  the  reliability  growth  program,  and 
some  unsubstantiated  "rules-of- thumb"  for  establishing  the 
initial  conditions  of  the  growth  models. 
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1 . 2  OBJECTIVES 

The  purpose  of  the  Reliability  Growth  Prediction  Study 
was  to  develop  a  procedure  for  predicting  the  reliability  growth 
which  could  be  expected  in  an  equipment  development  program 
based  on  the  characteristics  of  the  equipment  undergoing  relia¬ 
bility  growth  testing  and  the  attributes  of  the  development 
program.  As  the  study  proceeded,  this  objective  focused  on 
assessing  the  applicability  of  currently  accepted  reliability 
growth  measurement  models  (i.e.,  Duane,  AMSAA)  to  the  reliabil¬ 
ity  growth  prediction  problem.  Additional  objectives  included 
the  development  of  guidelines  for  (1)  identifying  what  type  of 
electronic  equipment  programs  are  most  amenable  to  imposition 
of  the  reliability  growth  philosophy  and  (2)  specifying  relia¬ 
bility  growth  in  equipment  development  contracts. 
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1 . 3  APPROACH 

To  accomplish  the  above  objectives,  TASC  first  con¬ 
ducted  an  extensive  literature  review  of  both  candidate  relia¬ 
bility  growth  models  and  documented  case  histories.  A  survey 
of  electronic  equipment  development  programs  which  imposed 
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reliability  growth  testing  was  conducted.  For  these  programs, 
a  data  base  was  developed  which  included  for  each  program: 


©  Characterization  of  the  equipment 

•  Characterization  of  the  reliability  growth 
program  (environmental  profile  for  test¬ 
ing,  test  length,  etc.) 

•  Failure/time  histories  observed  in  the 
reliability  growth  test 

®  Identification  of  all  failure  modes  sur¬ 
faced  in  the  test  and  classification  as 
design-related,  manufacturing  process- 
related,  or  random 

»  All  corrective  actions  identified  as  a 
result  of  the  growth  test. 


The  data  base  encompassed  nine  different  electronic  systems, 
each  developed  by  a  different  contractor,  and  30  equipment 
items  (Line  Replaceable  Units).  All  were  avionics  development 
programs.  These  data  were  organized  into  a  data  base  struc¬ 
ture  suitable  for  statistical  analysis. 


The  first  step  of  the  analysis  was  to  examine  the 
failure/time  histories  using  statistical  methods.  These  analy¬ 
ses  included  the  following: 


•  Fitting  of  the  candidate  growth  models 

to  the  data  using  least-squares  techniques 

•  Testing  the  goodness-of- f it  of  each  model 
to  each  data  set 

•  Formulation  of  alternative  models  relat¬ 
ing  reliability  growth  model  parameters 
to  equipment  characteristics  and  program 
attributes 

•  Multiple  linear  regressions  to  identify 
the  best  combination  of  predictive  factors 
and  tests  of  statistical  significance. 
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These  statistical  analyses  of  the  failure/time  histories  were 
supplemented  by  less  rigorous  examinations  of  the  failure 
mode,  failure  classification  and  corrective  action  data  which 
drew  upon  the  insights  of  individuals  involved  in  the  develop¬ 
ment  of  the  equipment  and  the  conduct  of  the  reliability  growth 
test.  These  two  forms  of  analysis  provided  the  basis  for  an 
assessment  of  the  true  nature  of  the  reliability  growth  achieved 
in  each  program  and  the  equipment  characteristics  and  program 
attributes  which  appeared  to  be  most  strongly  related  to  the 
achieved  growth. 

The  analyses  and  assessments  above  were  the  basis  for 
developing  a  procedure  for  predicting  achievable  reliability 
growth.  Guidelines  for  identifying  equipment  development  pro¬ 
grams  most  suitable  for  reliability  growth  testing  were  pre¬ 
pared  which  illustrate  how  achievable  reliable  growth  and  the 
cost  of  conducting  a  growth  program  can  be  balanced  against 
downstream  life-cycle  cost  savings.  In  the  course  of  the  de¬ 
velopment  program  survey,  a  number  of  "lessons-learned"  were 
accumulated  relative  to  how  future  reliability  growth  testing 
could  be  improved  to  better  specify  and  assess  reliability 
growth.  These  lessons-learned  were  translated  into  a  set  of 
guidelines  pertaining  to  how  the  specification  of  reliability 
growth  in  equipment  development  programs  should  be  structured. 
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The  remainder  of  this  report  is  organized  as  follows.  £ 

Chapter  2  reviews  reliability  growth  models  and  their  applica-  Bp 

biiity  to  reliability  growth  prediction.  Chapter  3  describes  ^ 

the  results  of  the  development  program  survey  and  the  data  base  ^ 

it 

constructed  from  this  survey.  The  analysis  of  the  reliability  vj 

growth  data  is  presented  in  Chapter  4.  Chapter  5  describes  a  Ij 
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procedure  for  predicting  reliability  growth  in  terms  of  equip¬ 
ment  characteristics  and  program  attributes.  Guidelines  for 
selecting  equipments  which  are  promising  candidates  for  relia¬ 
bility  growth  testing  and  for  specifying  reliability  growth 
are  presented  in  Chapter  6.  A  summary  of  the  study  findings 
is  contained  in  Chapter  7. 
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RELIABILITY  GROWTH  MODELS 
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The  definition  of  reliability  growth  for  purposes  of 
this  study  is  the  positive  improvement  in  a  reliability  parame¬ 
ter  over  a  period  of  time.  A  genera], ly  accepted  metric  of 
reliability  is  Mean  Time  Between  Failure  (MTBF),  or  its  in¬ 
verse,  failure  rate.  A  reliability  growth  model  is  an  ana¬ 
lytic  relationship  describing  MTBF  as  a  function  of  time  and, 
possibly,  other  physical  and/or  empirical  parameters.  This 
relationship  can  be  continuous  (i.e.,  progressive  reliability 
improvement  as  time  accumulates)  or  discontinuous  (e.g. ,  char¬ 
acterised  by  discrete  incremental  improvements  at  specific 
points  in  time),  or  a  combination  of  both.  As  some  form  of 
reliability  growth  model  must  underly  a  prediction  methodol¬ 
ogy,  a  survey  was  performed  to  identify  candidate  models  and 
their  suitability  to  prediction.  Initially,  emphasis  was 
placed  on  the  Duane  and  the  Army  Material  Systems  Analysis 
Activity  (AMSAA)  models,  because  these  are  the  most  widely 
recognized  and  commonly  used  models.  Some  difficulties  in 
applying  these  two  models  to  the  prediction  problem  resulted 
in  the  exploration  of  alternative  models  and  techniques,  in 
particular,  the  IBM  model. 


* 


2.1 


DUANE  MODEL 


J.T.  Duane  (Ref.  1),  in  1964,  discovered  that  an  em¬ 
pirical  linear  relationship  existed  between  the  logarithm  of 
cumulative  failure  rate  and  the  logarithm  of  cumulative  test 
time  for  various  equipments  that  had  been  through  engineering 
development.  Codier  (Ref.  2),  in  1968,  translated  Duane's 
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postulate  into  one  describing  the  association  of  the  logarithm 
of  cumulative  MTBF  to  the  logarithm  of  cumulative  test  time. 
Using  this  approach,  the  value  of  the  slope  of  the  line  de¬ 
scribing  this  interdependence  was  called  the  growth  rate  a. 
Higher  values  of  a  were  considered  representative  of  rapid 
growth  while  lower  ones  were  indicative  of  poorer  development 
efforts . 


When  using  the  Duane  approach,  cumulative  MTBF  is 
plotted  vs  cumulative  test  time  on  log-log  paper  and,  if  the 
data  appear  linear,  the  postulate  may  apply.  The  instantane¬ 
ous  MTBF  is  derived  as  the  cumulative  MTBF  divided  by  (1-a)  as 
shown  in  Section  2.1.1.  Figure  2.1-1  is  a  Duane  plot  for  an 
airborne  radar  showing  cumulative  MTBF  versus  test  time  and 
the  instantaneous  MTBF  line,  which  runs  parallel  to  it. 
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2.1.1  Mathematical  Description 

Underlying  the  Duane  model  is  the  hypothesis  that  as 
long  as  reliability  improvement  efforts  continue,  the  empiri¬ 
cal  relationship  in  Eq.  2.1-1  describes  the  change  in  cumula¬ 
tive  failure  rate  over  time. 


where 


XCUM  =  KT 


(2.1-1) 


=  cumulative  failure  rate 

Irl 

K  =  some  constant  which  Duane  felt  "will 
depend  on  equipment  complexity,  design 
margin,  and  design  objective  for 
reliability" 

T  =  cumulative  operating  or  test  time 

a  =  parameter  describing  rate  of  change  in 
MTBF  in  some  broad  sense. 


Inverting  Eq.  2.1-1  yields 


MTBFC  =  i  Ta 


(2.1-2) 


where 


MTBF^-.  =  cumulative  mean  time  between  failure  (MTBF). 


The  following  two  relationships  also  hold  when  fail¬ 
ure  data  are  plotted 


LCUM  T 


MTBFc  =  £ 


(2.1-3) 


(2.1-4) 


where 
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N  =  number  of  failures  observed  in  T  hours  of 
operation  or  test 


Using  Eq.  2.1-3,  the  current  or  instantaneous  MTBF  at 
time  T,  denoted  by  MTBF^ ,  cai  be  derived  as  follows: 


N  -  X 

T  CUM 


N  =  TA 
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N  =  TKT'a  ( from  Eq .  2.1-') 


N  =  KT 


i  _ 
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g  =  (l-«)  KT'" 


\(T)  =  (1-a )  KT' 
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MTBFr 
MTBFI  =  (T^T 


(2.1-5) 


The  straight  line  nature  of  the  logarithm  of  cumula¬ 
tive  MTBF  when  plotted  against  cumulative  operating  time  is 
apparent  by  taking  logarithms  of  both  sides  of  Eq.  2.1-2. 


log  MTBFC  =  log  ^  +  a  log  T 


2.1.2  Applicability  to  Prediction 


(2.1-6) 


The  Duane  model  is  a  two  parameter  model,  the  two 
parameters  being  a  and  K.  Therefore,  to  use  this  model  as  a 
basis  for  predicting  the  reliability  growth  which  could  be 
expected  in  an  equipment  development  program,  procedures  must 
be  defined  for  estimating  these  two  parameters  as  a  function 


of  equipment  characteristics  an.  .  program  attributes.  It  is 
important  to  note  that  each  of  these  parameters  is  empirical 
in  nature,  i.e.,  while  they  can  be  estimated  for  a  given  data 
set  using  curve- fitting  methods,  there  exists  no  underlying 
theory  for  the  Duane  model  which  would  provide  a  basis  for  a 
priori  estimation  of  a  and  K. 

There  does  exist  a  considerable  body  of  historical 
data  indicating  observed  values  of  a  in  prior  programs.  This 
experience  has  at  least  established  boundaries  for  a ,  namely 
that  it  ranges  from  a  minimum  value  of  0.1  to  a  maximum  value 
of  0.7.  However,  other  than  the  observation/assertion  that 
where  it  falls  in  this  range  is  somehow  related  to  the  "aggres¬ 
siveness"  of  the  contractor's  reliability  improvement  program, 
there  is  no  basis  for . estimating  a  in  advance.  Instead,  the 
only  apparent  grcundrule  is  to  base  an  advanced  estimate  of  a 
on  prior  corporate  experience  with  similar  equipments. 

The  parameter  K  is  even  more  ambiguous.  A  cursory 
examination  of  Eq.  2.1-6  reveals  that  1/K  is  something  of  an 
intercept  point  with  respect  to  the  log- log  plot  of  cumulative 
MTBF  versus  time.  In  particular,  at  T  =  1  nour,  MTRF^  =  1/K. 
Hence,  Tr  could  be  interpreted  as  the  inverse  of  the  cumulative 
MTBF  after  one  hour  of  operation.  This  is  a  meaningless  quan¬ 
tity  for  prediction  pi’rposes  prior  to  testing;  there  is  no 
reason  to  postulate  that  the  MTBF  at  one  hour  is  any  different 
from  the  MTBF  at  two  hours  although  the  Duane  model  implies 
otherwise . 

Hence,  a  principal  difficulty  in  using  the  Duane  model 
for  prediction  is  where  to  initialize  the  growth  curve.  In  most 
applications  of  the  Duane  model,  the  initialization  point  is 
determined  afterward.  That  is,  the  failure  versus  time  data 
are  laid  down  on  a  log-log  scale  and  the  initialization  point 
is  self-determined  based  on  where  the  best-fitting  line  falls. 
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One  generally  applied  "rule-of-thumb”  is  that  the 
ordinate  (MTBF)  should  be  initialized  at  10%  of  the  MIL-HDBK- 
217  predicted  MTBF.  An  open  question  is  what  is  the  abscissa 
value  (initial  time)  corresponding  to  this  initial  MTBF.  Clear¬ 
ly,  it  is  not  zero  as  Eq.  2.1-2  is  undefined  at  t  =  0.  An 
examination  of  Duane  plots  of  actual  data  in  the  literature 
(Refs.  2-4)  reveals  initialization  points  ranging  from  t  =  1  hour 
to  t  -  100  hours.  This  uncertainty  is  extremely  significant 
when  using  r.he  Duane  model  for  predictions  as  the  following 
simple  example  will  demonstrate. 

Suppose  that  a  system  with  a  predicted  MTBF  of  100  hours 


is  subjected  to  a  reliability  improvement  program  during  whim 
2000  hours  of  testing  will  be  performed.  Suppose,  also,  that 
the  initial  MTBF  is  10%  of  the  predicted  MTBF  (MTBF^  =  10  hours) 
and  that  it  is  somehow  known  that  a  reliability  growth  rate 
(or)  of  0.5  can  be  achieved  during  the  test.  Then  if  it  is 
assumed  that  the  initial  time  is  one  hour,  the  MTBF  at  the 
conclusion  of  testing  is 

MTBF2000  =  10  x  ( 2000/1 )° ‘ 5 
~  447  hours 

However,  if  the  time  is  initialized  at  100  hours,  then 

MTBF2000  =  10  x  (2000/100)°' 5 
=  45  hours 

Clearly,  the  Duane  model  is  highly  volatile  relative  to  this 
initialization  assumption,  an  undesirable  characteristic  for  a 
predictive  model. 
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The  above  discussion  is  not  meant  to  completely  dis¬ 
parage  the  Duane  model,  but  only  to  point  out  that  to  use  it 
for  prediction,  it  is  necessary  to  have  an  initial  set  of  time 
and  failure  data  to  initialize  the  model  and  estimate  the 
parameters.  Once  initialized,  the  model  has  proven  to  be  very 
useful  for  predicting  subsequent  reliability  growth.  However, 
as  illustrated  in  Section  4.1,  it  is  quite  difficult  to  estimate 
the  parameters  in  the  absence  of  such  initial  data,  i.e.,  based 
solely  on  equipment  characteristics  and  program  attributes. 


2.2  AMSAA  MODEL 

Crow  (Ref.  5)  approached  the  reliability  growth  prob¬ 
lem  by  stating  that,  within  a  particular  phase  of  a  development 
program,  the  failure  rates  would  change  over  time  as  prototype 
items  were  redesigned  or  reworked  because  of  problems  uncovered 
during  test.  An  assumption  is  that  each  test  phase  possesses 
a  different  level  of  growth,  depending  on  the  test  and  fix 
philosophy  during  that  test.  Crow  showed  that  if  this  were 
the  case,  conditions  would  exist  whereby  the  phenomena  could 
be  described  by  a  nonhomogeneous  Poisson  process. 

2.2.1  Mathematical  Description 

Within  a  particular  test  phase  design  modifications 
would  occur  at  cumulative  test  times  S, ,  S~,...S.,  as  shown  in 
Fig.  2.2-1,  with  i=4.  It  is  assumed  that,  even  though  there 
may  be  more  than  one  prototype  on  test,  the  basic  configura¬ 
tions  of  the  equipment  are  the  same,  therefore  the  failure 
rate  between  modifications  is  constant.  Let  represent  the 
constant  failure  rate  during  the  i*"*1  time  period  S^] 

between  modifications,  as  in  Fig.  2.2-2.  Based  on  the  constant 
failure  rate  assumption,  the  number  of  failures,  N^,  has  the 
Poisson  distribution  with  mean  \.(S.-S.  ,  ).  That  is, 
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Figure  2.2-1  Phase-by-Phase  Reliability  Growth 


Prob  (JT  =  n) 
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(2.2-1) 


Let  N(t)  be  the  total  number  of  system  failures  by 
time  t.  When  the  failure  rate  is  constant,  N(t)  follows  a 
homogeneous  Poisson  process  with  mean  A.t.  When  the  failure 
rate  changes  with  time  then,  under  certain  conditions,  N(t)  is 
said  to  follow  a  nonhomogeneous  Poisson  process.  Jn  this  par¬ 
ticular  situation,  N(t)  would  follow  such  a  process  with  mean 
value  function 


e(t) 


/  P(y)dy 
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(2.2-2) 
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Figure  2.2-2  Failure  Rates  Between  Modifications 


where 

p(y)  -  ^■i  >  y±e  si>  • 

Therefore,  for  any  t, 


Prob[N( t)=n] 


[ 0 ( t ) ]ne"6(t) 


n  i 


(2.2-3) 


where  n  =  0 ,  1 ,  2 , .  .  . 

The  AMSAA  model  assumes  that  p(t)  may  be  approximated 
by  the  parametric  form  p(t)  =  \pt^  ^  (Fig.  2.2-3),  t>0,  A>0, 
p> 0 ,  which  is  recognized  as  a  Weibull  failure  rate  function. 


I 


PHASE  1  j  PHASE  2 

t 


PHASE  3 


Figure  2.2-3  Parametric  Approximation  to  Failure  Rates 

Between  Modifications 


This  implies  that  the  mean  number  of  failures  by  time  t  is 
0(t)=\t^.  The  function  (p(t))  ^  =  (\pt^  =  m(t)  represents 

the  current  or  instantaneous  MTBF  of  the  system  at  time  t. 

2.2.2  Applicability  to  Prediction 

The  Duane  and  AMSAA  approach  are  related  in  that  they 
make  use  of  the  underlying  observed  linear  relationship  between 
the  logarithm  of  cumulative  MTBF  and  that  of  cumulative  test 
time,  but  the  Duane  is  empirical  only  and  does  not  provide  a 
capability  to  test  whether  the  change  in  MTBT  observed  over  time 
is  significantly  different  from  what  might  be  seen  due  to  random 
error,  whereas  the  AMSAA  model  allows  for  such  assessments. 
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The  AMSAA  approach  allows  for  development  of  hypothesis  testing 
procedures  to  determine  growth  presence  in  the  data  ({3<1  imply¬ 
ing  growth  in  MTBF,  {3=1  implying  constant  MTBF,  {3>1  implying 
wearout).  Confidence  bounds  on  the  parameter  {3  can  also  then 
be  calculated. 

However,  relative  to  applying  the  AMSAA  model  for 
prediction,  the  same  problems  exist  as  with  the  Duane  model 
(see  Section  2.1.2).  That  is,  while  it  is  possible  to  esti¬ 
mate  the  parameters  X  and  {3  for  a  given  set  of  failure- time 
data  using  statistical  procedures,  it  is  difficult  to  relate 
these  parameters  to  equipment  characteristics  or  program  attri¬ 
butes  a  priori  in  the  absence  of  such  data. 

2.3  IBM  MODEL 

The  IBM  model  (Ref.  6)  is  based  on  the  assumption 
that  at  any  point  in  time  there  are  two  types  of  equipment 
failures : 

•  Random  failures  which  occur  at  a  constant 
failure  rate,  X 

o 

•  Non-random  design,  manufacturing  and 
workmanship  defects. 

It  is  further  assumed  that  the  random  failure  rate  component 
is  always  present  at  the  same  level  in  the  equipment  and  cannot 
be  reduced  through  testing  and  corrective  action.  The  non- 
random  component,  on  the  other  hand,  is  progressively  reduced 
as  testing  is  conducted,  the  defects  are  surfaced,  and  correc¬ 
tive  actions  identified. 
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2.3.1  Mathematical  Description 


The  IBM  model  assumes  that  the  number  of  non- random 
defects  present  in  the  system  at  the  start  of  testing  is  fixed, 
but  unknown.  If  N(t)  denotes  the  number  of  non-random  defects 
remaining  after  t  hours  of  testing,  it  is  assumed  that  the 
rate  of  change  of  N(t)  with  respect  to  test  time  is  proportional 
to  the  number  of  remaining  defects,  that  is 
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(2.3-1) 


Thus , 


-K0t  +  C 
N(t)  =  e  ^ 


(2.3-2) 


The  number  of  defects  at  time  zero  is  a  constant,  namely 

N(0)  =  eC  (2.3-3) 

which  is  denoted  by  K-^ . 

it  follows  that  if  V(t)  is  defined  to  be  the  expected 
cumulative  number  of  failures  (random  and  non-random)  after  t 
hours  of  testing,  then 

-K9t 

V(t)  -  At-t-K,  (1-e  1  )  (2.3-4) 

o  i 

The  overall  failure  rate,  A(t),  is  then  given  by 

-K  *■ 

A(t)  =  ^  Aq  +  K^e  2 "  (2  3-5) 
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Thus  as  t  grows  large,  the  non-random  defect  component  becomes 
small  and  A(t)  approaches  Aq  as  expected. 


2.3.2  Applicability  to  Prediction 


s? 

&*> 


Because  of  the  nature  of  the  relationship  in  Eq.  2.3-4, 
the  IBM  model  is  not  as  conducive  to  the  plotting  of  failure 
data  as  are  the  Duane  and  AMSAA  models.  However,  by  manipulat¬ 
ing  the  terms  of  F.q.  2.3-5  and  taking  logarithms,  the  following 
relationship  can  be  derived: 


In  [A(  t)  -  \  )  =  InK-j  +  lnK2  -  K^t 


(2.3-6) 


Based  on  this  expression,  the  non -random  component  of  the  in¬ 
stantaneous  failure  rate  would  plot  as  a  straight  line  on  a 
log-linear  scale.  This  means  that  if  one  possesses  a  set  of 
failure  data  for  which  it  is  possible  to  discriminate  random 
failures  and  non-random  defects  (i.e.,  as  a  consequence  of  the 
failure  analyses),  then  the  parameters  Aq,  K-^  and  K2  could  be 
estimated  by  the  following  plotting  procedure: 


fev 


1. 

2. 


Group  the  non-random  defects  into  discrete 


intervals  of  time  (0 


b-j^ )  >  , 


b-i )  > 


Estimate  the  instantaneous  non-random 

failure  rate  for  each  time  interval  as 

A.  =N./(t.-t.  ,)>  where  N.  is  the  number 
i  iii-1  i 

of  non-random  failures  observed  in  time 

interval  i 

Plot  the  estimates  of  A^  versus  time  on  a 

log-linear  scale  and  fit  a  straight  line  to 
the  data  points 


4.  The  negative  slope  of  the  line  is  the  esti¬ 
mate  of  K2 

5.  The  point  at  which  the  line  intercepts  the 

ordinate,  say  y  ,  is  equal  to  K-,  .  Thus 

Kx  =  y0/K2 


6. 


Estimate  A.  as  M/t  where  M  is  the  total 
o 

number  of  random  failures  observed  in  test 
time  t. 


Of  course,  if  one  could  not  discriminate  between  random  and 
non- random  failures  in  the  data  set,  then,  in  theory,  the  three 
parameters  of  the  model  could  still  be  estimated  using  itera¬ 
tive  procedures.  However,  a  very  large  data  set  --  larger 
than  usually  would  be  generated  is  development  testing  --  would 
be  required  to  estimate  the  parameters  with  fidelity. 

Although  it  has  not  been  as  widely  utilized,  nor  its 
validity  clearly  established,  the  IBM  model  has  distinct  advan¬ 
tages  as  a  reliability  growth  prediction  tool  relative  to  the 
Duane  and  AKSAA  models.  These  characteristics  are  as  follows: 


It  distinguishes  between  random  and  non- 
random  failures  and  thus  permits  consid¬ 
eration  of  corrective  actions  (the  only 
way  by  which  real  reliability  growth  is 
achieved)  in  the  data  analysis 

Its  parameters  are  not  completely  empiri¬ 
cal;  i.e.,  they  have  a  physical  signifi¬ 
cance  and  it  is  thus  possible  to  formu¬ 
late  hypotheses  relating  the  parameters 
to  equipment  characteristics  and  program 
attributes . 


To  illustrate  the  latter  point,  consider  the  following  plaus¬ 
ible  conjectures  relative  to  estimating  the  three  parameters  a 
priori,  before  any  testing  is  performed: 


A  ,  being  the  random  failure  rate  compo¬ 
nent,  is  equivalent  (or  proportional)  to 
the  predicted  failure  rate  of  the  system 
per  MIL-HDBK-217 
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K^,  the  number  of  non- random  defects 

li-Iil  tially  resident  in  the  equipment,  is 
proportional  to  \Q  but  scaled  ^Qwn  t 

account  for  equipment  maturity  and  other 
reliability  progrim  elements  imposed  to 
eliminate  or  min:  mize  defects 

K2»  the  rate  at  which  defects  are  sur¬ 
faced  during  testing,  is  related  to  the 
conditions  (e.g.  ,  time  compression.,  envi¬ 
ronmental  stresses  imposed)  of  the  test. 


The  major  point  here  is  that  if  the  IBM  model  can  be  demonstrated 
to  validly  represent  the  reliability  growth  process,  then  proce¬ 
dures  for  estimating  its  parameters  for  prediction  purposes  can 
be  based  on  a  combination  of  statistical  methods  and  common- 
sense  engineering  reasoning,  ra.ther  than  statistical  techniques 
alone . 


2.4  OTHER  RELIABILITY  GROWTH  MODELS 


.  vv  V 


There  are  many,  many  other  forms  of  mathematical  models 
for  reliability  growth  that  have  been  proposed  in  the  litera¬ 
ture.  These  include  the  whole  family  of  discrete  growth  models, 
which  treat  reliability  growth  in  the  context  of  a  series  of 
trials.  One  example  is  the- Auto-Regressive- Integrated-Moving- 
Average  (ARIMA)  model  developed  by  Singpurwalla  (Ref.  7).  The 
ARIMA  model  is  a  time  series  approach  to  modeling  the  change 
in  MTBF  over  time.  It  essentially  looks  at  the  sequence  or 
series  of  MTBF  readings  over  time  and  derives  a  statistical 
relationship  between  the  MTBF  at  a  given  point  in  the  sequence 
and  the  previous  known  MTBFs.  The  model  can  be  used  to  track 
the  MTBF  over  time  and  to  predict  the  (k+1)  MTBF  given  the 
known  previous  MTBFs.  Statistical  algorithms  are  available 
which  will  automatically  calculate  the  values  of  the  time  series 
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parameters  and  thus  completely  specify  the  ARIMA  model  for  a 
given  sequence  of  MTBFs. 


Other  discrete  growth  models  are  described  in  Appen¬ 
dix  B  of  MIL-HDBK-189 .  While  some  analyses  based  on  the  ARIMA 
model  were  conducted  in  the  course  of  this  study,  it  was  found 
that  it,  and  the  other  discrete  growth  models,  were  not  gener¬ 
ally  suitable  for  the  type  of  failure  data  generated  in  the 
equipment  development  programs  included  in  the  survey.  In 
particular,  the  data  were  representative  of  a  single  test- 
analyze-fix  program  for  each  equipment  item,  as  opposed  to  a 
series  of  trials;  hence,  the  continuous  growth  models  were 
deemed  more  appropriate. 

MIL-HDBK-189  also  describes  various  other  continuous 
growth  type  models  which  were  not  evaluated  in  this  study.  A 
previous  RADC  reliability  growth  study  (Ref.  8),  conducted  by 
Hughes  Aircraft  Company,  evaluated  six  of  these  models  by  fit¬ 
ting  each  of  them  to  a  significant  number  of  data  sets.  The 
major  findings  from  the  Hughes  study,  which  are  relevant  to 
the  current  study,  are  repeated  below: 


•  Although  the  Duane  model  was  seldom  the 
best  fitting  model,  it  almost  always  fit 
the  data 

•  The  IBM  model  fit  airborne  data  the  best 

•  Each  of  the  other  four  models  was  found  to 
be  the  best  fit  to  the  data  for  specific 
combinations  of  environment,  equipment  type 
and  aggressiveness  of  reliability  program. 


In  summary,  the  study  did  not  identify  that  any  particular  model 
is  "best"  across  the  board,  but  that  the  Duane  model  is  general¬ 
ly  applicable  and  that  the  IBM  model  is  a  viable  candidate  for 
airborne  equipment,  which  is  the  type  of  equipment  analyzed  in 
the  current  study. 
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DEVELOPMENT  PROGRAM  SURVEY 


A  survey  was  conducted  of  historical  equipment  develop¬ 
ment  programs  in  which  reliability  growth  testing  was  performed, 
The  programs  were  reviewed  and  analyzed  to  identify  the  degree 
of  reliability  growth  that  was  achieved  during  test,  as  well 
as  the  equipment  characteristics  and  program  attributes  which 
appeared  to  enhance  reliability  growth  during  development  and 
those  which  appeared  to  have  a  negligible  effect.  This  chapter 
describes  the  development  program  survey  in  terms  of  the  pro¬ 
grams  included  and  the  data  base  constructed  to  support  the 
analysis.  Chapter  4  describes  the  analyses  of  the  program 
data. 


3 . 1  LITERATURE  SEARCH 

Initially,  a  review  of  the  reliability  literature  was 
conducted  to  identify  documented  data  sources  and  potential 
sources  for  the  acquisition  of  data.  This  search  was  effec¬ 
tive  in  identifying  equipment  programs  for  which  reliability 
growth  testing  had  been  performed  and  the  findings  from  analy¬ 
sis  of  the  reliability  growth  data,  but  unsuccessful  in  acquir¬ 
ing  the  raw  data  generated  during  the  program.  Generally, 
the  reliability  growth  data  presented  in  the  literature  con¬ 
sist,  at  best,  of  Duane  plots  of  failure  versus  time  data  re¬ 
sulting  from  the  test.  While  useful  information,  such  data 
sets  were  not  judged  to  be  appropriate  for  inclusion  in  this 
study  because  it  was  not  possible  to  ascertain  what  sort  of 
screening,  or  censoring  of  the  data  had  occurred,  nor  was  visi¬ 
bility  into  random  versus  non-random  failures  and  corrective 
actions  incorporated  achievable.  follow-up  with  the  authors 
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did  not  produce  the  original  data  for  reasons  of  corporate 
sensitivity  or  loss  of  the  data  over  time. 

Nevertheless,  the  literature  survey  was  of  value  in 
that  it  revealed  various  trends  of  thought  relative  to  relia¬ 
bility  growth  testing,  the  degree  of  acceptance  of  different 
growth  models,  as  well  as  some  of  the  pitfalls  involved  in 
analyzing  reliability  growth  data. 


3 . 2  PROGRAMS  SURVEYED 

The  programs  included  in  the  survey  were  programs  in 
which  TASC,  as  an  independent  advisor  to  the  program  office, 
acquired  all  relevant  data  generated  in  the  course  of  the  re¬ 
liability  growth  program,  and  conducted  analyses  of  the  data 
to  support  program  decisions.  These  programs  are  summarized 
below. 

3.2.1  Omega  Navigation  Set 

The  Omega  Navigation  Set  program  was  conducted  to 
develop  and  acquire  radio  navigation  equipment  to  replace  the 
outdated  LORAN-A  system  on  the  Air  Force's  C-130  series  of 
aircraft.  It  was  a  pioneering  program  for  its  time  in  that  it 
was  one  of  the  first  programs  to  emphasize  life-cycle  cost  as 
a  major  decision  criterion  and  in  its  use  of  the  Reliability. 
Improvement  Warranty. 

The  Omega  program  was  also  unique  in  its  imposition  of 
reliability  improvement  "esting  while  still  in  a  competitive 
phase.  In  particular,  pre-^^oduction  prototypes  from  each  of 
three  manufacturers  competing  for  the  production  contract  were 
subjected  to  a  Government-conducted  Combined  Stress  Reliability 


Test  (CSRT)  performed  in  an  environmental  test  chamber  at  the 
Air  Force  Flight  Dynamics  Laboratory.  Each  set  was  subjected 
to  stresses  in  this  test  designed  to  surface  failure  modes  and 
enable  the  respective  manufacturer  to  identify  and  incorporate 
corrective  action.  The  set  consisted  of  three  Line  Replaceable 
Units  (LRUs) 

®  Receiver-Processor  Unit 

•  Control-Display  Unit 

©  Antenna  Couple -Unit. 

TASC  established  and  maintained  a  data  base  of  all  failures 
observed  in  test,  including  the  time  of  failure,  equipment 
configuration  at  failure  and  subsequent  failure  resolution. 

3.2.2  Common  Strategic  Doppler 

The  Common  Strategic  Doppler  (CSD)  program  developed 
a  Doppler  navigation  capability  for  the  Air  Force's  KC-135  and 
B-52  aircraft  series.  The  CSD  consisted  of  two  LRUs: 

•  Doppler  Velocity  Sensor 

•  Ground  Speed  Drift  Indicator. 

As  for  the  Omega  program,  reliability  improvement  testing  was 
conducted  during  the  competitive  phase.  Two  sets  from  each  of 
two  manufacturers  were  subjected  to  a  Combined  Environment 
Reliability  Test  (CERT),  again  performed  in  the  Flight  Dy¬ 
namics  Laboratory  environmental  chamber. 

The  CERT  was  somewhat  different  from  the  Omega  CSRT 
in  that  rather  than  being  an  accelerated  test  (i.e.\  more  severe 
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than  the  mission  environment) ,  it  was  designed  to  simulate  the 
mission  environment.  Thus,  the  observed  reliability  in  test 
would  be  representative  of  that  to  be  expected  in  the  field. 
However,  it  could  also  be  considered  to  be  a  reliability  growth 
test  in  that  the  two  contractors  had  technical  representatives 
on-site  to  witness  the  test  and  return  failed  units  for  analysis 
and  corrective  action.  A  total  of  1700  hours  was  accumulated  for 
one  contractor  and  2500  hours  for  the  other.  TASC  maintained  a 
comprehensive  data  base  of  all  test  events  and  performed  inde¬ 
pendent  reliability  and  reliability  growth  assessments. 

3.2.3  Offensive  Avionics  System 

The  Offensive  Avionics  System  (OAS)  was  developed  to 
replace  the  aging  Bomb  Navigation  System  in  the  B-52  aircraft. 

The  overall  OAS  program  actually  entailed  seven  equipment  devel¬ 
opment  programs,  each  with  a  different  contractor.  The  programs 
were  for  the  following  equipments; 

•  Attitude  Heading  Reference  System 

•  Electronic  Altimeter  Set 

•  Control  Display  Set 

•  Radar  Set  Group 

•  Data  Transfer  Set 

•  Avionics  Control  Unit 

•  Interface  and  Control  Units. 

A  total  of  23  LRUs  comprised  the  OAS. 

The  program  was  conducted  under  a  compressed  schedule 
which  included  concurrency  of  Full  Scale  Engineering  Develop¬ 
ment  and  production.  Because  of  the  reliability  risk  inherent 
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in  concurrency,  an  Independent  Review  Team  evaluated  the  pro¬ 
gram  and  recommended  reliability  improvement  testing.  In  re¬ 
sponse,  the  program  office  embarked  on  a  Test-Analyze-Fix  (TAF) 
program  for  the  OAS.  The  testing  was  conducted  in  environmental 
facilities  at  each  of  the  respective  contractor  plants.  In 
most  cases,  two  test  articles  for  each  LRU  were  included  in 
the  TAF  program;  in  a  few  cases  four  were  tested.  The  target 
was  to  accumulate  2400  hours  of  test  on  each  equipment;  the 
actual  test  times  were  close  to  this  target,  some  being  higher 
and  some  lower. 

TASC,  under  contract  to  the  Program  Office,  established 
and  maintained  an  OAS  TAF  data  base  and  performed  an  independent 
assessment  of  the  reliability  growth  achieved  for  each  equipment 
(Ref.  9).  The  OAS  TAF  data  base  is  the  most  recent  and  compre¬ 
hensive  of  the  programs  surveyed.  To  illustrate,  Table  3.2-1 
indicates  the  depth  of  data  collected.  Therefore,  in  the  sub¬ 
sequent  analyses  this  set  of  data  was  used  as  the  principal 
source . 


3,3  DATA  BASE  ORGANIZATION 

The  data  acquired  from  the  development  programs  were 
organized  into  a  data  base  to  support  the  analyses.  The  data 
were  assembled  by  system  (manufacturer)  and,  within  each  sys¬ 
tem,  by  Line  Replaceable  Unit.  A  total  of  twelve  systems  and 
30  LRUs  were  included  as  listed  in  Table  3.3-1.  Two  sets  of 
data  were  constructed  for  each  system  and  LRU: 

•  Equipment  characteristics  and  program 
attributes 

•  Reliability  growth  test  data  (failure 
times,  failure  classification,  correc¬ 
tive  action). 
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TABLE  3.2-1 

OAS/TAF  DATA  ELEMENTS 


DATA  ELEMENT 


DESCRIPTION 


Identification  of  (JUT 


Serial  Number 


Operating  Time 
Cycle  Time 
Test  Conditions 
Discrepancy 
Maintenance  Activity 


Malfunction 


Maintenance  Time 


Contractor 

Representative 


Serial  number  of  operatmg/failed  LRU 

Total  on-hours  of  the  LRU  and  of  the  LRU  type  (especially  at  time  of  failure) 

Time  into  the  cycle  when  a  discrepancy  or  failure  is  noted 
Environmental  conditions  at  the  time  of  a  discrepancy  or  failure 
Observed  symptom,  test  operator  actions  and  BIT  indications 

Description  of  all  maintenance  actions  leading  to  on-site  repair  of  the  LRU,,  identify 
removal/replacement  of  any  subassembly  (SRU),  module,  and/or  component,  include  nomencla¬ 
ture  (item  name)  and  serial  number  of  replacement  item  Also  identify  any  adjustments 
performed  and  relationship  to  failed  item. 

Identify  the  function  or  parameter  within  the  LRU  that  was  performed  incorrectly  or  not- 
at-all  as  a  direct  result  of  the  item  failure,  and  its  relation  to  the  ooserved  symptoms 
If  the  failure  was  discovered  during  performance  verification  tests  and  not  during  func¬ 
tional  checkout/operation,  identify  why  BIT  did  not  detect  the  failure 

The  following  maintenance  times  shall  be  recorded. 

a)  on-site  time  when  bench  checkout  (repair)  was  initiated,  b)  on-site  time  when  repair 
was  completed,  c)  on-site  time  when  performance  verification  tests  were  successfully 
completed. 

Name  and  signature  of  contractor  representative  submitted  report. 


Failure  Analysis  As  a  minimum,  the  following  information  shall  be  provided: 


Module/Circuit/Comoonent  Failure  -  The  faulty  module/circuit  component  shall  be  identified 
to  the  extent  corresponding  to  depot-level  maintenance  and  to  the  degree  that  corrective 
action  is  proposed/performed  (i  e  ,  repair  of  part  vs  circuit/module  redesign) 

Cause  of  Failure  -  Identify  the  relationship  between  the  failure  and  the  environmental, 
operating  conditions  at  the  time  of  failure;-  include  pertinent  variations  in  other 
circuits  which  could  have  caused  the  failure  in  the  failed  circuit,  provide  sufficient 
detail  to  substantiate  the  failure  mechanism  (i.e.,  overdrive,  overvoltage,  overtempera- 
ture/excessive  heat  dissipation  -  as  relates  to  environmental  conditions) 

Corrective  Action  -  Identify  the  performed  and/or  proposed  corrective  action  to  avoid 
recurrence  of  the  failure.  Identify  how  the  modification  is  to  enhance  the  reliability 
design  of  the  failed  item. 


3.3.1  Equipment  Characteristics  and  Program  Attributes 

Electronic  equipments,  when  entering  full  scale  engi¬ 
neering  development,  possess  varying  levels  of  design  maturity, 
i.e.,  some  parts  or  assemblies  represent  proven  technology,  while 
others  represent  state-of-the-art  advancement.  The  device  it¬ 
self  may  be  highly  complex  and  intended  to  perform  a  complicated 
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TABLE  3.3-1 

SYSTEMS  AND  LRUS  COMPRISING  DATA  BASE 
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SYSTEM/LRU 

USAGE 

A* 

Provide  aircraft  attitude  and  heading  data 

A1 

Attitude  data  supply 

A2 

Data  conversion 

B 

Provides  measure  of  absolute  altitude  from  0-5000  feet 

B1 

Transmitter/Recsiver 

B2 

Data  converter  to  absolute  height 

C 

Provide  information  on  weapon  status 

Cl 

Input  transformation  for  video  display 

C2 

Records  and  produces  mission  data  on  35-millimeter  film 

C3 

Digital  computer,  command,  query,  record  weapon 
information 

C4 

CRT  display 

C5 

Keyboard  for  data  entry  into  system 

D 

Radar 

D1 

Switch  assembly 

D2 

Receiver/Transmitter 

E 

Receives  and  transmits  data,  stores  data  on  mag  tapes 

El 

Selects  processor,  programs  for  load 

E2 

Tape  transport 

F 

Two  processors,  aircraft  navigation  computational 
capability 

G 

Provides  necessary  data  conversions 

G1 

Main  interface  to  radars 

G2 

Interface  unit 

G3 

Interface  to  weapon  systems 

G4 

Interface  to  computational  subsystems 

G5 

Controls  for  weapon  launch 

G6 

Control  computer 

G7 

Controls  critical  navigation 

G8 

Provides  signal  interface  between  system  and  weapons 

H 

Navigation  equipment 

HI 

Receiver  processor  unit 

H2 

Control  display  unit 

I 

Navigation  equipment 

11 

Receiver  processor  unit 

12 

Control  display  unit 

J 

Navigation  Equipment 

J1 

Receiver  processor  unit 

J2 

Control  display  unit 

K 

Doppler  navigation 

L 

Doppler  navigation 

•Alphabetic  characters,  e.g.  A,  refer  to  systems. 
Alphabetic  characters  followed  by  numbers  refer  to  LRUs. 
A1  means  LRU  1  of  system  A. 
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array  of  mission  functions,  or  somewhat  simple  in  design  with  a 
minimal  set  of  performance  requirements.  The  usage  time  period 
may  be  brief,  but  with  environmental  variability,  or  long,  but 
under  relatively  benign  operational  surroundings.  It  may  be 
powered  in  a  cyclical  manner,  or  run  continuously  over  constant 
time  periods.  Any  combination  of  these  characteristics ,  i.e., 
maturity,  complexity,  environment,  and  usage  are  possible.  Each 
particular  mixture  will  influence,  among  other  things,  testing 
philosophy,  failure  isolation  and  identification,  and  design 
change  implementation.  This  implies  that  reliability  growth 
will  also  be  affected. 

Whatever  the  characteristics  of  the  equipment,  system 
development  from  a  cost/engineering/test  point  of  view  must  be 
managed  to  facilitate  eventual  entrance  into  the  military  inven¬ 
tory.  Resources  (time,  money,  manpower)  are  allocated  for  test¬ 
ing  environments,  total  test  duration,  and  parts/subassembly/ 
assembly  screening.  These  allocations  will  affect  the  type  of 
tests  and  the  time  involved  in  corrective  action  implementation, 
and,  hence  reliability  growth.  Funding  levels,  testing  types 
and  parts  screening  are  labeled  as  program  attributes. 

Information  on  each  system/LRU  was  organized  by  char¬ 
acteristics,  attributes,  and  specifications.  These  summaries, 
called  profiles,  were  arranged  as  follows: 

1.  Characteristics 


A. 

Usage 

intended 

use  of  equipment 

B. 

Environment 

using  MIL 

-STD  217D  definitions,  i.e., 

aircraft 

uninhabited  fighter,  etc. 

C. 

Complexity 

measured 

by  looking  at  no.  of  shop 

replaceable  units  (SRUs)  and  equip¬ 
ment  cost 
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D.  Maturity 


II.  Attributes 


A.  Type  Test 


B.  Stress 

Environment 


C.  Reliability 
Funding  Level 


D,  Parts  Control 


III.  Specifications 


reflected  by  the  %  of  off-the-shelf 
technology  (whether  parts  or  designs) 
used  by  the  system 


identifies  the  type  of  test  con¬ 
ducted,  whether  test-analyze-fix 
(TAF),  operational,  field  develop¬ 
ment,  etc,  along  with  testing  envi¬ 
ronments,  levels,  temperatures,  and 
conditions 

specifically  labels  test  environment 
as  one  intended  to  stimulate  failure 
or  simulate  operational  conditions 

percentage  of  total  contract  award 
dollars  clearly  identified  for  reli¬ 
ability  test  and  evaluation 

quality  assurance  program  in  place, 
if  any,  prior  to  start  of  test 


A.  MIL-STD  785B  tasks  imposed  contractually  or,  if  pro¬ 
gram  was  prior  to  MIL-STD  785B,  reliability-centered 
contractual  requirements 

B.  MIL-STD  217  prediction 

C.  Growth  test  planning  -  usage  of  MIL-STD-1635  or 
MIL-HDBK-189 


The  following  is  representative  of  such  a  profile. 


Program  C 


Characteristics 


A.  Usage 


Perform  data  insertion,  control 
supervision,  and  recording  of 
information 


B.  Environment  -  Aircraft  inhabited  bomber 


C.  Complexity  - 
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# SRU’s 

COST 

•: 

System 

84 

255K 

t 

t 

Cl 

24 

114K 

C2 

10 

47K 

i 

i 

C3 

23 

64K 

4 

C4 

18 

24K 

\ 

C5 

4 

6K 

H 

u 

1 

ity  - 

£ 

V 

V 

V 

% 

Off  Shelf 

Components 

s 

A 

R 

System 

Cl 

C2 

C3 

C4 

C5 


26-50 

10-25 

51-75 

10-25 

51-75 

51-75 


fl 

>4 


Attributes 

A.  Type  Test  -  TAF  (contractually  required  for 
2400  hours) 

-  Cycles  defined  as  follows: 


Equipment 

Temp . 
(°C) 

Cycle  on  Time 
(hrs ) 

Max  Vib. 
g/rms 

Vib.  on  Time 
(min .  ) 

1 

Cl 

-54 

,  7  C 

4.0 

6.0 

340 . 0 

C2 

-54 

»  5o 

^.0 

6.0 

340.0 

C3 

-54 

,  71 

4.0 

6.0 

340.0 

,*v 

C4 

-54 

,  55 

4.0 

6.0 

440.0 

C5 

-54 

,  55 

4.0 

6.0 

440.0 

» 

srr 

-  Chamber  test 

£ 

3. 

Stress 

Environment  - 

Stimul ation 

\  *  ’ 

s 

C. 

R^Iiabi 

iity  Funding 

Level  -  >1% 

of  contract  award 

•4  _ 

D. 

Quality 

Assurance  ?r 

ogram  in  conjunction  with  - 

k  % 

ii 
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MIL-Q-9858A 

MIL-C-45662A 

MIL-I-6870D 


MIL-STD-1535A 
MIL'- STD- 1520A 


(Quality  Program  Reqmt's) 
(Calibration  System  Reqmt's) 
(Nondestructive  Inspection  Program 
Requirements  for  Aircraft  and 
Missile  Materials  and  Parts) 
(Supplier  Quality  Program  Require¬ 
ments  > 

(Corrective  Action  and  Disposition 
System  for  Non-Conforming  Material) 


HI. 


Quality  Assurance  Status  Report  Required 
Specifications 

A.  Failure  Review  and  Corrective  Action  System  in 
place  prior  to  start  of  test 

B.  MIL-STD-217 -B  Prediction  (MTBF)  - 


Pred. 

Reqm 1 t 

System 

359.0 

250.0 

Cl 

1091.0 

1000.0 

C2 

3752.0 

3300.0 

C3 

1480.0 

1400.0 

C4 

6150.0 

3800.0 

C5 

3316.0 

2900.0 

C.  Growth  planning  done  in  accordance  with 
MIL-STD-1635 


3.3.2  Reliability  Growth  Test  Data 

For  each  system,  as  well  as  tor  each  LRU  comprising 
the  system,  a  data  set  representing  the  results  of  the  relia¬ 
bility  growth  test  was  constructed.  An  extract  from  the  data 
base  presenting  these  data  sets  for  two  LRUs  is  provided  in 
Table  3.3-2  in  order  to  illustrate  the  format.  Each  row  entry 
corresponds  to  a  discrete  test  failure  event  with  the  follow¬ 
ing  data  contained  for  each  failure  event: 


>  -v  vviwi/  /  >  v  .*  »• 


•  REPORT  NO.  -  Identifies  the  number  of 
the  contractor  failure  report  written  as 
a.  result  of  the  failure  event  (available 
in  hard  copy) 

•  FLAG  -  Identifies  the  failure  mechanism 
as  distinct  or  as  identical  to  one  al¬ 
ready  observed  in  test 

9  CAUSE  -  Identifies  the  failure  as  being 
due  to  a  design  defect  (D) ,  a  manufac¬ 
turing/workmanship  defect  (M) ,  non- 
relevant  (NR)  or  random  (R) . 

The  FLAG  and  C^USE  entries  are  very  important  relative  to  re¬ 
liability  growth  assessment.  The  FLAG  permits  discrimination 
of  new  failure  modes  from  repetitive  failures.  For  example, 
the  first  seven  failure  events  for  LRU  G1  were  all  due  to  the 
same  design  defect.  The  CAUSE  permits  discrimination  between 
random  failure  (CAUSE  =  R)  and  defects  which  could  be  elimi¬ 
nated  through  corrective  action  (CAUSE  =  D  or  M) .  Of  the  fif¬ 
teen  failure  events  observed  in  testing  of  LRU  G1 ,  one  was 
non-relevant  (e.g. ,  due  to  a  chamber  problem),  seven  were  due 
to  a  single  design  defect,  two  were  due  to  a  second  design 
defect,  two  were  due  to  a  manufacturing  defect  and  three  were 
random . 

The  important  point  to  note  is  that  to  merely  plot 
the  cumulative  MTBF  versus  the  cumulative  test  time  as  a  method 
for  assessing  reliability  growth  (which  would  indicate  a  sig¬ 
nificant  MTBF  growth  of  from  five  to  167  hours)  would  be  to 
mask  out  some  very  relevant  information,  namely  the  number  of 
failure  modes  and  correctable  defects  surfaced. 


DATA  ANALYSIS 


4.1  OBSERVATIONS 

Before  presenting  the  various  analyses  performed  on 
the  development  program  survey  data  base,  some  less  rigorous, 
but  germane,  findings  will  be  presented.  These  observations, 
which  are  a  result  of  both  TASC's  first-hand  participation  in 
the  reliability  growth  testing  for  the  programs  surveyed  and 
of  the  data  analysis  performed,  will  serve  to  illustrate  some 
of  the  peculiarities  and  potential  misinterpretations  of  relia¬ 
bility  growth  data. 

4.1.1  Time  Constraints 

It  is  difficult  to  argue  the  point  that  reliability 
improvement  results  only  from  the  identification  of  defects 
and  the  incorporation  of  corrective  actions.  In  an  ideal  world, 
testing  would  be  stopped  as  soon  as  a  failure  is  observed  and 
not  resumed  until  the  failure  is  analyzed  and  a  corrective 
action  identified  and  incorporated  into  the  test  article. 

This,  of  course,  is  impractical  and  not  cost-effective.  Each 
of  the  programs  surveyed  by  TASC  was  under  some  sort  of  pres¬ 
sure  to  complete  testing  within  a  reasonable  period  of  calendar 
time.  Testing  couldn't  be  nalted  whenever  a  failure  occurred. 

As  a  consequence,  corrective  actions  were,  at  best,  incorporated 
into  test  articles  several  hundred  operating  hours  after  the 
failure  orginally  surfaced,  and  often,  were  not  incorporated 
at  all  but  instead  were  identified  for  incorporation  au.ing 
production.  Hence,  the  results  of  corrective  actions  were 
only  minimally  reflected  in  the  test  data. 
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This  issue  has  been  belabored  in  the  literature 
(Ref.  10,  for  example).  The  question  is  whether  reliability 
growth  as  measured  on  test  data  is  a  consequence  of  true  relia¬ 
bility  improvement  of  the  system,  or  just  reflects  the  burning 
in  of  the  serialized  units  undergoing  test.  The  argument  goes 
that  a  given  serialized  unit  will  have  a  certain  number  of  bad 
parts  due  to  random  lot  selection  and  as  these  parts  fail  and 
are  replaced  with  better  parts,  the  apparent  reliability  im¬ 
proves.  There  is  no  doubt  that  this  effect  is  present,  to 
some  degree,  in  any  set  of  data  from  a  reliability  growth  pro¬ 
gram  performed  in  FSED  on  a  finite  number  of  test  articles. 

4.1.3  Cumulative  Data 

The  standard  procedure  for  using  the  Duane  approach 
to  track  reliability  growth,  is  to  plot  cumulative  MTBF ,  (i.e., 
cumulative  time/cumulative  failures)  on  a  log-log  scale.  Work¬ 
ing  exclusively  with  cumulative  data  can  be  quite  misleading 
when  measuring  reliability  growth.  A  simple  example  illustrates 
this  point.  Specifically,  suppose  that  the  system  undergoing 
test  is  characterized  by  having  a  number  of  initial  defects 
(i.e.,  infant  mortality),  but  a  constant  failure  rate  after 
removal  of  these  defects.  Five  failures  are  observed  in  the 
first  100  hours  of  testing,  and  a  failure  is  observed  every 
100  hours  thereafter  (to  convey  the  point,  randomness  in  the 
test  data  is  eliminated).  Table  4.1-1  illustrates  how  the  data 
from  1000  hours  of  testing  would  be  presented  cumulatively. 

Clearly,  the  data  imply  that  reliability  growth  is 
taking  place,  even  though  the  true  MTBF  is  constant  at  100  hours 
after  the  first  100  test  hours.  Furthermore,  if  these  data  are 
plotted  in  a  log-log  scale,  they  appear  to  lie  on  a  straight 
line  with  a  slope  (a),  of  0.43. 
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TABLE  4.1-1 

EXAMPLE  OF  CUMULATIVE  DATA 


CUMULATIVE 

HOURS 

NUMBER  OF 
FAILURES 

CUMULATIVE  MTBF 
(HOURS) 

100 

5 

20 

200 

6 

33 

300 

7 

43 

400 

8 

50 

500 

9 

56 

600 

10 

60 

700 

11 

64 

800 

12 

67 

900 

13 

69 

1000 

14 

71 

The  point  of  this  example  is  that  it  is  not  necessarily 
remarkable  --  nor  a  validation  of  the  Duane  model  --  that  fail¬ 
ure  data  tends  to  appear  linear  when  plotted  on  a  log- log  scale. 
A  high  incidence  of  early  failures,  and  the  progressive  dimin¬ 
ishing  of  their  impact  through  cumulative  plotting,  may  very 
well  be  the  underlying  cause  for  such  behavior,  particularly 
when  dealing  with  small  data  sets  (e.g.  ,  less  than  fifteen 
failures) . 


4.2  STATISTICAL  ANALYSES  OF  GROWTH  DATA 

The  first  method  of  investigation  utilized  in  this 
study  was  an  analysis  of  the  failure-time  history  data  generated 
in  the  different  reliability  growth  testing  programs.  These 
analyses  were  based  only  on  the  temporal  characteristics  of 
the  failure  data,  and  did  not  discriminate  failures  by  type  or 
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corrective  actions.  Specifically,  the  data  set  for  each  equip¬ 
ment  consisted  of  the  sequence  (i\,T^),  ‘ 

where  T.  is  the  cumulative  test  time  when  the  ith  failure  was 
1 

observed  and  T\/i  is  the  observed  cumulative  MTBF. 

4.2.1  Methodology 

Both  the  Duane  and  AMSAA  models  were  fit  to  the  data. 

An  International  Mathematical  and  Statistical  Library  (IMSL) 
regression  analysis  subroutine  was.  used  to  calculate  least 
squares  estimates  of  the  growth  model  parameters,  e.g. ,  a  and 
K  for  the  Duane  model. 

The  goodness-of-fit  was  then  assessed.  The  best  fit¬ 
ting  model  was  used  to  produce  a  series  of  predicted  cumulative 

A  A  A 

MTBFs,  i.e.,  MTBF-^,  MTBF2 ,  •••  MTBF^,  which  could  then  be  com¬ 
pared  to  the  observed  values  T-^/l ,  T2/2,  •••  T^/N .  Three  meas¬ 
ures  were  used  to  quantify  the  prediction  error.  These  error 
measures  are  given  in  Table  4.2-1.  The  criteria  used  to  deter¬ 
mine  significance  of  model  fit  were  the  Percent  Variation  Ex¬ 
plained  (Duane  model)  and  the  Cramer  Von  Mises  Statistics  (AMSAA 
model) . 


The  Duane  and  AMSAA  models  can  be  used  to  predict  the 
instantaneous  MTBF  at  the  end  of  a  test  interval.  In  order  to 
measure  the  error  associated  with  these  instantaneous  MTBF 
predictions,  the  failure  time  history  data  were  divided  into 
equal  width  subintervals  and  the  instantaneous  Duane  and  AMSAA 
predictions  computed  at  the  midpoint  of  the  subintervals  were 
compared  to  the  observed  MTBF  computed  over  each  subinterval . 
Figure  4.2-1  illustrates  this  technique.  The  bar  heights  cor¬ 
respond  to  the  cumulative  MTBF  over  the  corresponding  interval 
of  test  time.  The  bar  lengths  are  chosen  heuristically  as  the 
minimum  length  required  to  divide  the  total  test  time  into  sub 
intervals  such  that  each  subinterval  has  at  least  one  failure 
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R  MEASURES 


VARIABLE 

GENERIC  NAME 

FUNCTIONAL  FORM 

INTERPRETATION 

R1 

Average  Absolute 
%  Error 

...  .  ®BVHIBFt  100 

1  Z-4  MTBF.  N 

t=l 

A  model  fits 
well  if  R.,£25 

R2 

Relative  Variability 

N 

Y)  -  2 

~  (MTBF  -MTBF  )/N-2 

■n  _  t***  i  L.  U 

R2  "  N  9 

(MTBFt-MTBF)  /N-l 

t=l 

Low  values  of 

R^  indicate  a 

good  fit. 

USE 

Mean  Squared  Error 

N 

MSE  =  (MTBFt-MTBFt)2/N 

t=l 

Low  values  of 
MSE  indicate  a 
good  fit. 

■■sa ■spates 


(i.e.,  the  cumulative  MTBF  over  each  subinterval  is  not  infi¬ 
nite).  The  error  in  the  instantaneous  MTBF  prediction  for  the 
ith  subinterval  (i=l,...,M;  M=total  number  of  subintervals)  is: 


e,  =  (observed  MTBF  over  subinterval)  - 

X 

(model-calculated  MTBF  interval  midpoint) 


The  error  measures  R^  ,  R2 ,  and  MSE  defined  in  Table  4.2-1  were 
applied  to  the  instantaneous  errors  (e^;  i=l,...,M)  to  assess 
the  overall  prediction  error  and  significance  of  fit. 


Next,  it  was  attempted  to  correlate  the  estimated 
growth  rates  (i.e.,  a  for  the  Duane  model  and  p  for  the  AMSAA 
model)  to  the  equipment  characteristics  and  program  attributes 
listed  in  Section  3.3.1.  A  series  of  multiple  linear  regression 
analyses  were  performed  using  various  combinations  of  the  charac 
teristics  and  attributes  as  the  independent  variables  and  the 
estimated  growth  rate  as  the  dependent  variable.  Both  single 
and  multiple  variable  linear  forms  were  examined,  although  the 
multiple  variable  forms  examined  were  primarily  limited  to  two- 
variable  models.  The  general  form  of  the  two-variable  model  was 


or 


Plh 


P2X2 


(4.2-1) 


where 

a  =  Growth  rate  for  the  equipment  as  estimated 
by  the  Duane  model  fit 

X-,  ,Xo  =  Independent  variables  (characteristics/ 
attributes ) 

P. ,?2  =  Constants  yielding  the  best  relation  of  or  to 
1  X|  and  X2  using  a  least-squares  criterion 


An  IMSL  program  was  used  to  accomplish  the  regression.  For 
each  independent  variable  (or  pair  of  independent  variables) 


1)^1 . . 


^  wv 


I 


tested,  the  goodness  of  the  regression  model  was  measured  by  the 
percentage  variation  in  the  growth  rate  parameter  explained  by 
the  model,  an  outyut  of  the  IMSL  program. 


ran 

s$ 

$- 

W 

& 


4.2.2  Results  of  Statistical  Analysis 


KV 


iv 


Duane  Model  Fits  -  The  Duane  model  was  fit  to  all  equip¬ 
ments  with  failure- time  histories  in  the  data  base.  Table  4.2-2 
displays  the  estimates  for  the  Duane  intercept  (log(l/K))  and 
slope  (a)  for  each  equipment.  A  key  indicator  of  the  regression 
fit  to  the  Duane  relationship  between  the  cumulative  MTBF  and  the 
cumulative  test  time  is  the  Duane  Percent  Variation  explained 
column  in  Table  4.2-2.  The  higher  the  variation  explained,  the 
greater  the  amount  of  variation  in  the  data  that  can  be  accounted 
for  by  the  fitted  Duane  model  and,  hence,  the  more  significant 
the  fit. 


The  following  observations  can  be  drawn  from 
Table  4.2-2: 


•  After  eliminating  the  data  sets  with  less 
than  five  data  points,  approximately  half 
the  data  sets  display  a  reasonable  fit 
(%  variation  >  75%) 

•  With  one  exception  (Equipment  C2),  the 
equipments  displaying  a  negative  growth 
rate  have  either  a  small  number  of  data 
points  or  do  not  display  a  good  Duane 
fit  (%  variation  <  25%) 

•  For  those  equipments  which  both  have  a 
significant  number  of  data  points  and 
display  a  good  fit,  the  growth  rate  ranges 
from  a  low  of  0.097  to  a  high  of  0.644. 


Vi 

cv 


Oc 

I 

»\L 
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The  general  conclusion  is  that,  while  there  are  some  anomalies,  Sy 

the  Duane  model  provides  a  reasonably  good  fit  to  most  of  the  yy 

> 

data  sets  and  the  estimated  growth  rates  are  within  the  gener-  of. 

.  155 

ally  accepted  range  of  0.1  to  0.7. 


TABLE  4.2-2 

DUANE  PARAMETER  ESTIMATES  FOR  EQUIPMENTS 


EQUIPMENT 

DUANE 

GROWTH  RATE 

DUANE 

INTERCEPT 

(LAMBDA) 

DUANE  % 
VARIATION 
EXPLAINED 

NO.  OF 
DATA 
POINTS 

Al 

.520 

2.578 

88.6 

3 

B1 

.663 

10.500 

97.3 

3 

B2 

-2.087 

22.388 

4.3 

7 

Cl 

.340 

2.538 

78.4 

21 

C2 

-.419 

8.300 

55.9 

20 

C3 

.486 

2.732 

74.7 

6 

C4 

.097 

4.254 

14.4 

31 

C5 

.644 

1.679 

97.4 

5 

D2 

.374 

1 . 664 

94.8 

31 

El 

.141 

2.977 

19.4 

50 

E2 

.474 

6.546 

87.9 

59 

E3 

.528 

1.965 

97.7 

7 

G1 

.594 

.398 

98.6 

14 

G2 

.572 

1.248 

96.1 

9 

G3 

-.253 

5.793 

7.5 

18 

G5 

- .  213 

6.546 

17.4 

9 

G6 

-4.056 

38.018 

96.5 

3 

G7 

.540 

.151 

90.3 

31 

G8 

-.174 

6.664 

23.5 

13 

HI 

.259 

.523 

47.0 

23 

H2 

-.024 

1.471 

.7 

21 

H3 

.368 

.855 

94.5 

11 

H4 

.251 

1.246 

55.4 

10 

11 

.196 

.694 

31.3 

16 

12 

.353 

.563 

48.3 

19 

13 

.585 

-.137 

98.2 

11 

14 

.416 

.572 

90.7 

9 

15 

.541 

-.016 

97.7 

4 

J1 

-.109 

2.714 

9.6 

16 

J2 

.237 

1.109 

53.6 

13 

J3 

.286 

1.605 

_ 

77.1 

11 

TABLE  4.2-3 

AMSAA  PARAMETER  ESTIMATES  EQUIPMENTS 


EQUIPMENT 

AM^AA 

GROWTH  RATE 

AMSAA 

LAMBDA 

CRAMER 

VON  MISES 
STATISTIC 

NO.  OF 
DATA 
POINTS 

Al 

.849 

.0040 

.098 

3 

B1 

.899 

.0027 

.140 

3 

B2 

5.779 

.0000 

.265 

7 

Cl 

1.049 

.0048 

.224 

21 

C2 

1.600 

.0001 

.046 

20 

C3 

1.408 

.0001 

.235 

6 

C4 

.852 

.0189 

.249 

31 

C5 

.703 

.0i00 

.080 

5 

D2 

.630 

.1669 

.105 

31 

El 

1.060 

.0119 

.122 

50 

E2 

.794 

.0882 

.446 

59 

E3 

.790 

.0123 

.144 

7 

G1 

.412 

.5543 

.050 

14 

G2 

.639 

.0616 

.03-6 

9 

G3 

.  640 

.1228 

.741 

18 

G4 

1.247 

.0001 

.094 

2 

G5 

.813 

.0159 

.215 

9 

G6 

5.917 

.0000 

.073 

3 

G7 

.501 

.6235 

.268 

31 

G8 

.947 

.0055 

.173 

13 

HI 

.723 

.5686 

.530 

23 

H2 

.579 

.9931 

.537 

21 

H3 

.693 

.2774 

.039 

11 

H4 

.652 

.3347 

.204 

10 

11 

.634 

.7940 

.667 

16 

12 

.778 

.2940 

.353 

19 

13 

.503 

.6956 

.059 

11 

14 

.542 

.5209 

.096 

9 

15 

.870 

.3404 

.090 

4 

J1 

.843 

.1744 

.253 

16 

J2 

.588 

.5344 

.194 

13 

J3 

1.115 

.0275 

.258 

11 

model  is  the  Cramer  Von  Mises  statistic.  Values  of  the  Cramer 
Von  Mises  statistic  less  than  0.28  generally  indicate  a  good 
fit  to  the  data. 
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Figure  4.2-2  AMSAA  Growth  Rates  for  ail  Equipments 

Based  on  the  Cramer  Von  Mises  statistic  as  the  crite¬ 
rion,  a  good  fit  was  observed  on  six  equipment  data  sets,  and 
a  marginal  fit  was  observed  on  eight  other  equipments.  As  was 

i 

the  case  with  the  Duane  model,  eome  equipments  exhibited  nega¬ 
tive  reliability  growth  (p  >  1),  but  in  ail  such  cases  either 
the  number  of  data  points  was  small  or  che  fit  v>as  not  signifi¬ 
cant  (or  both).  Fc1"  those  equipments  exhibiting  a  good  fit,  t 

the  AMSAA  growth  parameter  (p)  ranged  from  0.579  to  0.794. 

* 

> 

Prediction  Error  Analysis  -  The  accuracy  of  the  Fitted  ‘ 

Duane  and  AMSAA  models  as  predictors  of  reliability  growth  was  * 

tested  using  the  cumulative  and  instantaneous  error  analysis  •; 

techniques  described  in  Section  4.2.1.  The  results  of  the  ' 

cumulative  prediction  error  analysis,  in  terms  of  the  three 
error  measures  defined  in  Table  4.2-1,  are  presented  in  | 

Table  4.2-4.  The  equivalent  results  for  the  instantaneous  «• 
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TABLE  4.2-4 

CUMULATIVE  PREDICTION  ERROR  ANALYSIS 


SYSTEM 

DUANE 

AMSAA 

R] 

R2 

MSE 

R1 

R2 

MSE 

3 

0.883x10* 

0.204x10° 

0.123xl04 

NA 

NA 

NA 

82 

NF 

NF 

NF 

NA 

NA 

NA 

C 

0.135xl02 

0.392x10° 

0.657xl02 

NF 

NF 

NF 

Cl 

0  181xl02 

0.412x10° 

0.759xl03 

0.495xl02 

0.152x10* 

0.281xl04 

C2 

0 . 120xl02 

0.681x10° 

0.357xl04 

0.187x10* 

0.129x10* 

0.675xl04 

C  4 

NF 

NF 

NF 

NF 

NF 

NF 

D 

0.129xl02 

0.142x10° 

0.673xl02 

0.153xl02 

0. '222x10° 

0 . 105x 103 

D2 

0 . 102'<102 

0.800X10'1 

0.501xl02 

0.130xl02 

0 . 144x ; 0° 

0.903xl02 

E 

NF 

NF 

NF 

NF 

NF 

NF 

E1 

NF 

NF 

NF 

0.270xl02 

0.127x10* 

0.127xl03 

E2 

0 . 246v102 

0.439x10° 

0.158xl03 

NF 

NF 

NF 

£3 

0.102xl02 

0.135x10° 

0.292xl04 

0 . 565x 102 

0.412x10° 

0.891xl04 

F 

NF 

NF 

NF 

0.490xl02 

0.122x10* 

0.258xl05 

G 

0.159X102 

0.141x10° 

0.410x10* 

NF 

NF 

NF 

G1 

0.108xl02 

0.385x10'* 

0 . 115x 103 

0 . 19?x 102 

0.292x10'* 

0.875xl02 

G2 

0.131xl02 

0.923x10'* 

0.465xl03 

0.373xl02 

0.958x10'* 

0. 482xl03 

G3 

NF 

NF 

NF 

NF 

NF 

G5 

NF 

NF 

NF 

0 . 405x 102 

0.219x10* 

0.534xl04 

G7 

0.213xl02 

C.i62xlO° 

0 . 372x 102 

NF 

NF 

G8 

NF 

NF 

NF 

0.248xl02 

0.137x10* 

0.429xl04 

HI 

NF 

NF 

NF 

NF 

NF 

NF 

H2 

NF 

NF 

NF 

NF 

NF 

NF 

H3 

0 . 805x 101 

0.692x10'* 

0.974x10° 

0.222xl02 

0.328x10° 

0.462x10* 

H4 

0.134xl02 

0.841x10° 

0.383x10* 

0.318xl02 

0.238x10* 

O.lOSxlO2 

11 

NF 

NF 

NF 

NF 

NF 

NF 

12 

NF 

NF 

NF 

NF 

NF 

NF 

13 

0 . 107x 102 

0.566x10'* 

0.151x10* 

0  240xl02 

0.100x10° 

0 . 26£x 10* 

14 

C'.132xl02 

0.138x10° 

0.225x10* 

0.289xl02 

0.393x10° 

0.641x10* 

J1 

NF 

NF 

NF 

NF 

NF 

NF 

J2 

0.168xl02 

0.460x10° 

0  312x10* 

0 . 314x 102 

0.902x10° 

0.612<10* 

J3 

0 . 126x 102 

0.361-10° 

0. 112x 102 

_ _ 

NF 

---  - t-  - - r-* 

NF 

NTF 

NF  indicates  no  significant  fit 
NA  indicates  predictions  not  computable 
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prediction  error  analysis  are  presented  in  Table  4.2-5.  Both 
models  were  reasonably  accurate  predictors  of  cumulative  MT3F 
as  measured  by  the  R1 ,  R2 ,  and  MSE  values,  with  the  Duane  model 
being  more  accurate  across  the  board.  The  AMSAA  model  demon¬ 
strated  better  accuracy  for  predicting  instantaneous  MTBF . 


TABLE  4.2-5 

INSTANTANEOUS  PREDICTION  ERROR  ANALYSIS 


CVCTTM 

DUANE 

AMSAA 

A 

81 

82 

MSE 

. -R1 

82 

MSE 

£ 

£ 

B 

0.285xi02 

O.lllxlO1 

0.227xl05 

NA 

NA 

r . .  " 

NA 

• 

rm 

«p 

C 

0.457x10^ 

0.171X101 

O.lOOxlO1 

NF 

NF 

NF 

s'  f 

W" 

Cl 

0.716xl02 

0.147X101 

0 . 146x 103 

0.480xl02 

0.157X101 

0 . 156X10"5 

1 

C2 

0.478xl02 

0.144X101 

0.333xl05 

0.434xl02 

0.129X101 

0.299x10s 

D 

0.420xl02 

G.137X101 

0  - 190x 103 

0.445xl02 

O.llAxlO1 

0.l58xl03 

D2 

0.365xl02 

0.120X101 

0.157xl0S 

0.412xl02 

0 . 103x 101 

0.138x10s 

V 

El 

NF 

NF 

NF 

0.419xlC2 

0.141X101 

0.^83xl06 

E2 

0.935xlOz 

0.132X101 

0.726xl04 

NF 

NF 

NF 

$ 

E3 

0.785xl02 

0.313X101 

0.149xl06 

0.430xl02 

0.210X101 

0.994x’,05 

s 

F 

NF 

NF 

NF 

0.468xl02 

0.124X101 

0.313xl0S 

sV 

y 

G 

0 . 435x 102 

0.130X101 

0.444xl04 

NF 

NF 

NF 

it 

G1 

0 . 394x 102 

0 . 122x 101 

0.323xl05 

0.455xl0? 

0.917x10° 

0.263xl05 

G2 

0.440xl02 

O.llSxlO1 

0.229xl0S 

0.337xl02 

0.911x10° 

0.178x10s 

G5 

NF 

NF 

NF 

0 . 652x lo2 

0.130X101 

0.234xl0S 

,-i 

G7 

0.668xl02 

0.216X101 

0.184x10s 

NF 

NF 

NF 

if 

G8 

NF 

NF 

NF 

C.344x102 

0.966x10° 

0 . 108x 103 

H3 

0 . 373x 102 

0.114X101 

0.l43xl03 

0.399xl02 

0.107X101 

9. 134xl03 

x\ 

H4 

o.eioxio2 

0.199X101 

0.557xl02 

O.lOlxlO-5 

0.351X101 

0  933x 192 

■y  : 

13 

0.831xl02 

0.262X101 

0. 256xl03 

0 . 733x 102 

| 

0  208xl0] 

0 . 20'.x  10' 

W, 

14 

0.395xl02 

0 . 217x 101 

0.459x10s 

0.390xl02  ! 

I 

0.106X101 

0.223x:0S 

•-V 

J2 

0.624xio2 

0 . 348x 101 

0 . 117x io4 

0.591xl02 

0 . 173x 101 

0.580x10s 

J3 

0.7S2xl02 

0 . 178x 101  ; 

0  643xI0S  | 

i 

NF 

NF  j 

NF 

>; 

NF  indicates  no  significant  fit 
NA  indicates  predictions  not  computable 
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Growth  Rate  Prediction  -  Although  the  linear  regression 
analyses  to  correlate  observed  growth  rates  with  equipment 
characteristics  and  program  attributes  were  performed  for  both 
the  Duane  and  AMSAA  models,  findings  are  presented  herein  only 
for  the  Duane  model  because: 


•  rhe  reliability  growth  data  analyzed 
were  reflective  of  a  single  test  tor 
each  equipment  as  opposed  to  a  series  of 
test  phases 

•  The  findings  from  the  AMSAA.  regression 
analyses  were  consistent  with,  but  did 
not  augment,  those  from  the  Duane  model 
analyses . 


Thus,  the  focus  of  the  subsequent  discussion  is  on  the  Duane 
model  growth  parameter,  a,  and  the  factors  which  seem  to  ex¬ 
plain  its  observed  values  for  the  different  equipment  failure- 
time  histories. 


As  described  in  Section  4.2.1,  various  multiple  linear 
regression  models,  assuming  different  characteristics/attributes 
as  tne  independent  variables  and  a  as  the  dependent  variable, 
were  postulated,  the  regression  constants  estimated  and  then 
tested  for  the  degree  to  which  the  independent  variable(s) 
"explained"  the  variations  in  a  across  the  different  equipmenr 
data  sets.  As  is  usually  the  case  when  examining  real-world 
data  (as  opposed  to  data  generated  in  a  statistically  designed 
experiment)  ,  the  characteristics  of  the  data  were  not  ideal 
for  chis  type  of  analysis.  The  principal  data  limitations 
were: 

The  large  number  of  potential  independ¬ 
ent  variables  relative  to  the  number  of 
equipment  data  sets 


9 


•  Lack  of  sufficient  variability  across 
the  different  equipment  items  for  some 
of  the  candidate  independent  variables. 

Nevertheless,  the  data  base  was  believed  to  be  sufficiently 
robust  that  this  method  would  surface  any  factor,  or  pair  of 
factors,  which  were  obvious  explanatory  variables  relative  to 
other  factors. 

The  results  of  the  regression  analyses  did  not  produce 
any  single  linear  combination  of  parameters  which  explained 
more  than  50%  of  the  variability  in  the  Duane  growth  parameter 
estimates.  Hence,  no  such  model  was  identified  to  be  statis¬ 
tically  significant. 

Further  analyses  revealed  that  the  variability  was 
actually  due  to  factors  other  than  equipment  characteristics 
or  program  attributes.  in  particular,  the  influence  of  early 
failures  was  investigated.  (See  Section  4.1.3  for  a  discussion 
of  how  early  failures  can  influence  the  apparent  growth  rate). 
The  regression  analyses  were  conducted  using  time  to  first 
failure  (i.e.,  the  first  failure  occurring  in  the  growth  test) 
as  an  additional  candidate  independent  variable.  The  analyses 
yielded  an  extremely  insightful  result,  namely  that  the  best 
model  for  estimating  the  Duane  growth  rate  was 

a  =  -0.0028  x  FTF  +  0.00019  x  TTT  (4.2-1) 


where 

a  =  Duane  growth  rate  parameter 
FTF  =  First  time  to  failure 
TTT  =  Total  test  time. 
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From  a  regression  point  of  view,  this  model  explained  better 
than  87%  of  the  variability  in  the  Duane  growth  parameter 
estimates  across  the  data  sets  and  was  highly  statistically 
significant . 

Since  time  to  first  failure  is  more  a  single  data 
point  subject  to  randomness  than  it  is  a  predictable  quantity, 
identification  of  the  strong  relationship  of  it  to  a  did  not 
contribute  to  the  study  objective  of  relating  reliability  growth 
to  equipment  characteristics  and  program  attributes.  However, 
it  did  serve  to  explain  the  difficulties  in  trying  to  discern 
such  relationships  by  examining  historical  growth  curves.  It 
also  highlighted  the  pitfalls  involved  in  working  with  cumula¬ 
tive  data,  particularly  for  relatively  small  data  sets,  and 
the  need  to  pay  special  attention  to  the  influence  of  early 
failures  when  measuring  reliability  growth. 

Analyses  were  conducted  to  determine  whether  first 
time  to  failure  is  an  approximation  to  initial  MTBF  at  program 
start,  which  is  usually  estimated  as  10%  of  the  MIL-STD  217 
prediction.  Table  4.2-6  relates  these  concepts  by  displaying 
the  ratio  of > first  time  to  failure  to  the  MIL-STD  217  predic¬ 
tion.  Note  that  the  data  in  the  table  are  ranked  in  order  of 
ascending  MIL-STD  217  predictions.  Analyses  conducted  to  deter¬ 
mine  if  first  time  to  failure  could  be  statistically  related 
to  MIL-STD  217  prediction  led  to  inconclusive  results.  Fig¬ 
ure  4.2-3  displays  a  histogram  of  first  failure  time/MIL-STD  217 
prediction.  From  this  histogram  it  can  be  seen  that  the  first 
time  to  failure  cannot  be  estimated  as  10%  of  the  MIL-STD  217 
prediction.  The  variability  of  these  ratios  indicates  that 
the  average  percentage  is  closer  to  6%  of  the  MIL-STD  217  pre¬ 
diction  and  that  the  variability  is  significant. 


TABLE  4.2-6 

EQUIPMENT  FIRST  TIME  TO  FAILURES 


EQUIPMENT 

(FIRST  TIME  TO 
FAILURE/MIL- 21 7  PRED) 

FIRST  TIME 
TO  FAILURE 

TOTAL  NUMBER 
OF  FAILURES 

Cl 

.05500 

60.0 

21 

C3 

.11892 

176.0 

6 

D2 

.00660 

11.5 

31 

G8 

.13309 

288.0 

13 

C5 

.02654 

88.0 

5 

G7 

.00006 

.2 

31 

C2 

.07249 

272.0 

20 

B1 

.13997 

548.0 

3 

G1 

.00097 

4.9 

14 

Al 

.04151 

214.0 

n 

02 

.00224 

12.0 

9 

C4 

.01431 

88.0 

31 

El 

.00107 

7.2 

50 

G3 

.03150 

249.0 

18 

A2 

.18698 

1485.0 

1 

G5 

.03950 

316.0 

9 

G6 

.21409 

1836.0 

3 

G4 

.04689 

576.0 

2 

E2 

.00078 

10.8 

59 

D1 

.03597 

666 . 0 

2 

B2 

.06129 

1592.0 

7 

E3 

.00171 

68.4 

7 

Less  rigorous  analyses  of  the  data  did  yield  some 
first-level  relationships  between  reliability  growth  and  char¬ 
acteristics/attributes.  For  example,  equipment  maturity  was 
found  to  be  a  significant  factor.  The  equipment  data  sets 
which  had  a  small  number  of  failures  and/or  did  not  display  a 
good  Duane  model  fit,  tended  to  be  the  more  mature,  in  some 
cases  off-the-shelf,  items.  It  can  be  generally  concluded 
that  significant  reliability  growth  will  not  be  achieved  on 
such  items.  The  relationship  between  growth  rate  and  degree 
of  maturity  is  less  quantifiable,  due  to  the  difficulty  in 
defining  the  latter.  For  example,  one  such  definition  might 
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Figure  4.2-3  Histogram  of  First  Failure 

Time/MIL-217  Prediction 


be  percentage  off-the-shelf  technology  based  on  a  design  assess¬ 
ment;  however,  analyses  revealed  no  significant  relationship 
between  growth  rate  and  this  metric  as  applied  to  the  equip¬ 
ments  in  the  data  base. 


The  type  of  growth  test  conducted  was  also  determined 
to  be  an  influence  on  reliability  growth  rate.  Referring  to 
Table  4.2-2,  items  designated  by  the  letters  A  through  G  were 
subjected  to  a  Test-Analyze-Fix  program.  The  test  environment 
was  designed  to  stimulate  failures  and  hence  the  conditions  were 
accelerated  relative  to  the  mission  environment  (i.e.,  mission 
candidates  of  prolonged,  relatively  benign  operation  were  re¬ 
duced  or  eliminated).  The  items  designated  by  the  letters  H 
through  J  were  subjected  to  a  test  in  which  the  conditions  re¬ 
flected  a  simulation  of  the  mission  environment.  Eliminating 
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those  equipments  displaying  a  small  number  of  data  points  and/ 
or  an  insignificant  Duane  fit,  the  average  growth  rate  over 
the  equipments  subjected  to  the  accelerated  test  was  approxi¬ 
mately  0.5.  The  average  growth  rate  over  those  equipments 
subject  to  the  mission  simulation  type  test  was  0.35.  By  ex¬ 
trapolation,  and  recognizing  that  a  minimum  growth  rate  of  0.1 
is  possible  even  without  an  agressive  corrective  action  program, 
it  is  estimated  that  a  benign  test  (i.e.,  equipment  operation, 
but  limited  environmental  stresses)  would  yield  a  reliability 
growth  rate  on  the  order  of  0.2,  providing  the  contractor  is 
motivated  to  improve  reliability. 


I,  4 
«/• 


Growth  Model  Initialization  Considerations  -  The  Duane 
reliability  growth  models  fit  to  the  equipment  data  sets  were 
also  examined  from  the  standpoint  of  the  initialization  problem. 
Referring  to  Table  4.2-2,  the  relevant  initialization  parameter 
is  the  Duane  Intercept,  which  can  be  interpreted  as  the  loga¬ 
rithm  of  the  ordinate  (MTBF)  at  the  point  T  =  1  hour.  This 
being  a  somewhat  difficult  quantity  to  interpret  in  any  intui¬ 
tive  sense,  the  data  were  transformed  to  the  observed  initiali¬ 
zation  point  in  the  context  of  the  MIL-STD-1635  rule-of-thumb 
which  suggests  that  a  reliability  growth  curve  be  initiated  as 
follows  for  planning  purposes: 


•  Start  the  initial  MTBF  (MTBF  )  at  10%  of 
the  MIL-HDBK-217  prediction  ° 

•  Start  the  initial  test  time  (T  )  at 

100  hours,  or  50%  of  the  MIL-HDBK  pre¬ 
diction,  whichever  is  greater. 

50%  of  the  MIL-HDBK-217  prediction  was  the  larger  quantity  for 
all  equipments.  The  equivalent  initial  MTBFq  at  time  Tq  -  MTBF^/2 
was  derived  for  each  equipment  as  follows: 


MTBFQ  =  1/K  (MTBFp/2 ) 


(4.2-2) 


where  a  and  1/K  are  the  parameter  estimates  resulting  From  the 

JU 

Duane  model  fit  (Table  4.2-2  ).  Table  4.2-7  presents  the  re¬ 
sults  of  this  transformation  for  the  equipment  items  with  posi¬ 
tive  growth  rates  and  displaying  good  fits  to  the  Duane  model. 


Examination  of  Table  4.2-7  does  not  suggest  a  discern¬ 
ible  relationship  between  the  initial  MTBFq  (or  the  ratio  MTBFq/ 
MTBFp)  and  the  Duane  growth  rate  or  the  predicted  MTBF.  No 
intuitive  relationship  to  equipment  characteristics  was  identi¬ 
fied  either;  in  fact  one  intuitively  plausible  relationship 
was  somewhat  contradicted.  Specifically,  it  would  be  plausible 

to  assume  that  the  initial  MTBF  /MTBF  ratio  would  be  lower 

o  p 

for  less  mature  equipments.  However,  item  C3  was  one  of  the 
less  mature  equipment  items  in  the  data  base,  and  it  displayed 


TABLE  4.2-7 

TRANSFORMED  INITIAL  MTBF 


:fment 

DUANE  GROWTH 
RATE  (a) 

MTBF 

P 

T 

0 

(MTBFp/ 2) 

1/K 

MTBF 

0 

MTBFq/MTEF. 

(%) 

Cl 

0.34 

1091 

545 

12.165 

108 

10% 

C3 

0.486 

1480 

740 

15.36 

381 

26 

C5 

0 . 644 

3316 

1658 

5.36 

634 

■ 

19 

D2 

0.374 

1743 

871 

5.28 

66 

4 

E3 

0.582 

40000 

20000 

7.135 

1331 

3 

61 

0.594 

5066 

2533 

1.49 

105 

2 

G2 

0.572 

5359 

2679 

3.48 

318 

6 

G7 

0.540 

362.5 

1812 

1.16 

67 

2 

*The  Duane  Intercept  value  in  Table  4.2-2  is  actually  iog(l/K) 
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the  highest  MTBFq  (relative  to  MTBFp).  One  significant  obser- 
vaclon  that  can  be  made  is  that  while  the  MTBFo/MTBFp  displays 
considerable  variation ,  the  average  across  all  equipments  is 
approximately  9%.  In  a  sense,  these  data  suggest  that  the 
MIL-STD-1635  rule-of-thumb  is  not  all  that  bad,  in  the  absence 
of  any  other  information. 


4.2.3  Summarv  of  Findings 


The  results  of  the  analyses  of  the  equipment  relia¬ 
bility  growth  data  can  be  summarized  as  follows. 


•  Both  the  Duane  and  AMSAA  models  provided 
good  fits  to  the  data  sets,  with  the 
Duane  model  providing  significant  fits 
to  a  larger  number  of  equipment  data 
sets 

«  The  estimated  growth  rates  (excluding 
poor  fits)  were  within  the  historically 
observed  range  (0.1  <  a  <  0.7)  in  all 
but  a  single  case 

•  The  significant  influence  of  early  fail¬ 
ures  on  reliability  growth  rate  tended 
to  mask  out  any  influence  by  equipment 
characteristics  or  program  attributes 

•  Mature  equipments  displayed  little  or  no 
reliability  growth 

•  Equipments  subjected  to  a  TAF-type  test 
displayed  an  average  growth  rate  of  0.5; 
equipments  subjected  to  a  mission  simu¬ 
lation-type  test  displayed  an  average 
growth  rate  of  0.35;  it  was  extrapolated 
that  equipments  subjected  to  a  relatively 
benign  operational  test  would  display  a 
growth  rate  of  0.2 

•  The  MIL-STD-1635  reliability  growth  test 
initialization  conditions  were  found  to 
be  a  reasonable  rule-of-thumb  in  the 
absence  of  any  historical  data. 


However,  it  was  also  concluded  that  for  a  number  of  reasons 
(extended  burn-in  issues,  lack  of  discrimination  between  ran¬ 
dom  and  correctable  failures,  pitfalls  of  plotting  cumulative 
data),  the  Duane  growth  curve,  as  conventionally  applied,  does 
not  always  provide  an  accurate  representation  of  true  reliabil¬ 
ity  growth. 

4.3  CORRECTIVE  ACTION  ANALYSIS 

The  analyses  presented  in  Section  4.2  essentially 
treated  all  failures  observed  in  growth  testing  the  same.  In 
particular,  failures  which  resulted  in  subsequent  corrective 
action  by  the  contractor  were  not  distinguished  from  "random" 
failures.  Since  it  is  logical  to  argue  that  true  reliability 
growth  results  only  from  the  identification  and.  incorporation 
of  corrective  actions  (design  changes  or  manufacturing  process 
improvements),  additional  analyses  were  conducted  on  those 
equipment  data  sets  for  which  corrective  action  information 
was  available.  This  information  consisted  of  a  classification 
of  each  observed  failure  as  to  whether  the  failure  was  random 
or  whether  a  corrective  action  was  identified.  It  should  be 
noted  that  data  relative  to  time  of  incorporation  of  the  cor¬ 
rective  actions  into  the  units  under  test. were  not  generally 
available.  Hence,  there  was  no  means  of  validating  the  effec¬ 
tiveness  of  the  corrective  action. 

4.3.1  Failure  Mode  Classification 

For  discussion  purposes,  the  terminology  established 
by  Crow  in  Refs.  11  and  12  will  be  used.  Crow  discussed  a 
ratio  of  failure  rate  due  to  B  failure  modes  (Xg)  divided  by 
total  failure  rate,  where  total  is  defined  to  be  the  failure 
rate  due  to  A  modes  plus  failure  rate  due  to  B  modes  (X^+Xg) . 


An  A  failure  mode  is  one  where  no  engineering  fix  has  been 
assigned,  while  a  8  failure  mode  is  one  where  an  engineering 
or  manufacturing  process  fix  has  been  designated.  Mathemati¬ 
cally  this  ratio,  labelled  K,  is 


K  = 


lB 


hA  * 


(A. 3-1) 


From  a  practical  point  of  view,  K  represents  the  per¬ 
centage  of  failures  that  a  contractor  can  adequately  define 
and  identify  as  fixable.  In  a  sense  it  is  reflective  of  the 
way  in  which  a  contractor  works  the  problems  that  are  uncovered 
during  test.  If  many  of  the  failure  modes  that  are  encoun¬ 
tered  are  defined  as  random  and,  therefore,  have  no  corrective 
actions  assigned  to  them,  they  will  remain  inherent  within  the 
equipment.  The  device  reliability  will  be  affected  accordingly. 

Figure  4.3-1  is  a  histogram  of  the  observed  K-factors 
for  the  eqxiipment  data  sets.  It  can  be  seen  that,  for  the 
most  part,  the  K-facto>.  ranged  from  25%  to  70%.  The  low-end 
extreme  case  was  exhibited  by  two  extremely  mature  equipments 
which  exhibited  only  Type  A  failure  modes  during  the  test. 

The  high-end  extreme  case  was  exhibited  by  an  item  for  which 
the  observed  failures  were  dominated  by  recurring  failure  modes 
which  were  deemed  fixable. 


Because  of  the  wide  spread  of  the  K- factor  values  at 
the  item  (LRU)  level,  the  failure  data  were  aggregated  at  the 
manufacturer  level  to  gain  further  insight.  This  aggregation 
is  reasonable  since  it  is  the  manufacturer  who  is  analyzing 
and  classifying  the  failures,  as  well  as  identifying  the  cor¬ 
rective  actions.  Table  4.3-1  presents  the  aggregated  data  for 
six  programs.  The  data  in  Table  4.3-1  suggest  that  reliability 
growth,  as  measured  by  corrective  actions  identified  or  by 
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K-FaCTORS 

Figure  4.3-1  Histogram  of  K-Factors  for  Equipment  Data  Sets 

failure  modes  eliminated  througft  corrective  action,  is  related 
to  the  complexity  of  the  equipment  as  reflected  by  predicted 
MTBF .  The  MIL-HDBK-217  MTBF  prediction  is  a  meaningful  meas¬ 
ure  of  complexity,  representing  as  it  does  both  the  total  num¬ 
ber  of  parts  in  the  eouipment  and  the  complexity  of  the  parts 
themselves.  Clearly,  more  corrective  actions  were  identified 
for  these  subsystems  with  lower  predicted  MTBFs. 

However,  complexity  is  not  the  only  factor  influenc¬ 
ing  the  number  of  corrective  actions  identified.  In  order  to 
discern  other  influencing  factors,  the  data  were  normalized  to 
an  equivalent  complexity  by  multiplying  the  corrective  action 
count  by  the  ratio  of  the  predicted  MTBF  to  1,000  hours.  The 
normalized  data  are  shown  in  Table  4.3-2.  These  data  demon¬ 
strate  that  the  highest  level  of  correction  action  activity 


TABLE  4.3-2 

RELIABILITY  GROWTH  TEST  PROGRAM  DATA 
(NORMALIZED) 


SUBSYSTEM 

CHARACTERISTICS 

NORMALIZED 
NUMBER  OF 
CORRECTIVE  ACTIONS 

NORMALIZED  NUMBER 

OF  FAILURES 
ADDRESSED  BY 
CORRECTIVE  ACTIONS 

A 

Mature 

Technology 

5.5 

8.2 

B 

Mature 

Technology 

5.6 

11.3 

C 

New  Equipment; 
Several  JHUs 

9.7 

19.7 

D 

Partially  Mature 
Technology 

4.6 

11.9 

F 

Off-the-Shelf 
Prior  TAF 

1.1 

1.1 

G 

New  Equipment; 
Several  LRUs 

11.4 

46 

j 

was  generated  for  the  least  mature  equipments,  namely  Subsys¬ 
tems  C  and  G.  Similarly,  the  least  corrective  actions  were 
observed  on  the  most  mature  equipment  item  in  the  data  base, 
Subsystem  F.  which  was  not  only  an  off-the-shelf  item,  but  had 
also  been  subjected  to  a  Test-Analyze-Fix  ^TAF)  program  in  an 
earlier  aircraft  program. 


The  Type  A  (random)  failures  by  subsystem  observed  in 
test  are  presented  in  Table  4.3-3.  The  Type  A  MTBF  displayed 
in  the  test  is  calculated  as  the  test  time  divided  by  the  num¬ 
ber  of  Type  A  failures  observed.  What  is  significant  is  that 
the  Type  A  MTBF  is  proportional  (within  reasonable  statistical 
error  bounds)  to  the  MIL-HDBK-2.17  predicted  MTBF.  As  shown, 


w**  V*  x  ■ *>  *^V-V»V 
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TABLE  4.3-3 

RANDOM  FAILURE  COMPONENT  OF  TEST  DATA 


SUBSYSTEM 

TEST 

TIME 

(hrs) 

TYPE  A 
FAILURES. 
DURING  TEST 

TEST  MTBF 
(TYPE  A 
ONLY) 

PREDICTED 

MTBF 

RATIO 

(PREDICTED  MTBF/ 
IF ST  MTBF) 

. 

A 

B 

2416 

2488 

5 

6 

483 

415 

2038 

2837 

5.7 

6.8 

C 

2912 

69 

46 

359 

7.8 

* 

D 

2838 

30 

95 

664 

7.0 

F 

2331 

11 

212 

1079 

5.2 

i 

* 

G 

2718 

27 

101 

636 

S.3 

i 

I  ■ 


r.he  ratio  of  predicted  MTBF  to  Type  A  test  MTBF  was  consistent 
across  the  different  subsystems,  the  average  being  approxi¬ 
mately  6.5. 


The  fact  thaL  -Be  Type  A  test  MTBF  was  not  equivalent 
co  the  preciicted  MTBF  can  be  attributed  to  the  acceleration 
factor  associated  witn  the  TAF  test.  In  particular,  the  TAF 
tested  the  equipments  at  the  extremes  of  the  mission  environ¬ 
mental  profile,  and  reduced  or  eliminated  period-  of  reia^ 
tivtly  benign  operation.  As  a  result,  random  Type  A  failures 
occurred  at  a  rate  approximately  6.5  times  higher  than  would 
be  expected  in  the  normal  mission  environment,  which  the  MiL- 
HDBF-217  prediction  reflects.  The  consistency  of  the  ratio 
across  suosystems ,  all  of  which  were  subjected  to  equivalent 
TAF  conditions,  suggests  that  a  random  (Type  A)  component, 
which  is  rel ated  to  the  prediction exists  in  the  f a i lure 
data . 
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The  findings  that  Type  B  failures  are  related  to  com¬ 
plexity  and  maturity,  and  that  a  random  failure  component  re¬ 
lated  to  the  HIL-HDBK-217  prediction  exists,  suggest  that  the 
IBM  model,  described  in  Section  2.3,  might  be  a  valid  means  of 
representing  the  growth  data.  As  discussed  in  Section  2.3.2, 
this  model  has  distinct  advantages  relative  to  the  Duane  model 
for  prediction  purposes  in  that: 

9  It  allows  discrimination  between  Type  A 
and  Type  B  failures 

•  Its  parameters  are  more  interpretable  in 
a  physical  sense  than  are  the  Duane 
parameters . 

Investigation  of  the  IBM  model  is  also  suggested  by  Ref.  13. 

The  author  of  this  paper  applied  the  IBM  model,  as  well  as 
several  other  growth  models  including  the  Duane  model,  to  a 
ground  radar  in-house  test  program  and  concluded  that  "the  IBM 
model  is  highly  recommended."  These  considerations  motivated 
a  test  of  the  viability  of  the  IBM  model  for  reliability  growth 
prediction. 

The  data  base  did  not  support  a  completely  comprehen¬ 
sive  analysis  of  the  IBM  model.  Referring  to  Table  4.3-1,  it 
can  be  seen  that  only  two  subsystems  (C  and  G)  surfaced  a  suffi¬ 
cient  number  of  Type  B  failure  raocies  to  support  fitting  a  two- 

JL. 

parameter  model  to  the  data.  The  IBM  model  was  applied  to 
the  Subsystem  C  and  Subsystem  G  as  follows: 

^Including  \  ,  the  IBM  model  is  actually  a  three-pcrameter 
model;  however.  \  can  be  independently  estimated  by  extract¬ 
ing  the  Type.  A  failures  from  the  data  set. 
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Types  A  and  B  failures  were  distinguished 
based  on  the  failure  classifications  in 
the  data  base 


The  first  occurrence  of  each  Type  B  fail¬ 
ure  mode  was  identified,  subsequent  occur¬ 
rences  of  the  same  failures  were  screened 
from  the  data  base 


The  procedure  described  in  Section  2.3.2 
was  applied  to  estimate  the  parameters 
Kx  and  K2- 


The  results  of  the  IBM  model  fit  to  the  two  data  sets  are  shown 
in  Table  4.3-4. 


While  definitive  conclusions  cannot  be  drawn  from  an 
examination  of  only  two  data  sets,  the  viability  of  this  model¬ 
ing  approach  is  supported  by  the  fact  that  a  good  fit  was  ob¬ 
tained  for  these  two  sets.  It  is  also  significant  to  note 
that  the  fit  of  the  IBM  model  to  the  sample  data  set  in  Ref.  13 
resulted  in  estimates  of  the  parameters  K-^  and  K2  (K-^  =  29.44, 
K2  =  0.0Q03644)  which  are  of  the  same  magnitude  as  the  esti¬ 
mates  in  Table  A. 3-4. 


TABLE  4.3-4 

IBM  MODEL  PARAMETERS  FOR  SUBSYSTEMS 


SUBSYSTEM 


MODEL  PARAMETER 

A 

0 

Ki 

K2 

0.022 

32.3 

0.00041 

0.0049 

22.1 

0.00064 
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The  estimated  values  for  the  parameters  K-^  and 
provides  a  basis  for  developing  prediction  values  in  advance 
of  testing  based  on  engineering  considerations.  First,  con¬ 


sider  the  relation  of  to  predicted  MTBF  for  the  two  data 


sets : 


C: 

K,  x 

MTBF  = 

£ 

1 

P 

& 

G: 

K,  x 

MTBF  = 

& 

1 

P 

These  results  tend  to  confirm  the  intuitively  plaus¬ 
ible  assumption  that  the  number  of  Type  B  failure  modes  resi¬ 
dent  in  an  equipment  at  the  start  of  testing  (K^)  would  be 
proportional  to  complexity,  which  could  be  measured  by  pre¬ 
dicted  MTBF.  It  is  also  reasonable  to  assume  that  equipment 
maturity  and  other  reliability  program  elements  imposed  in  the 
development  program  would  influence  K^.  For  subsystems  C  and 
G,  it  was  estimated  that  these  considerations  accounted  for 
50%  of  the  Type  B  failures  having  been  removed  prior  to  the 
TAF  test. 


Next,  consider  the  factor  Since  subsystems  C  and 

G  were  subjected  to  very  similar  TAF  programs,  one  would  ex¬ 
pect  that  the  rate  of  surfacing  Type  B  failures  (K^)  would  be 
approximately  the  same.  Indeed,  the  difference  in  for  the 
two  subsystems  is  not  large.  A  possible  explanation  for  the 
difference  that  does  exist  is  that  the  manufacturer  of  Sub¬ 
system  G  was  the  prime  contractor  and  the  manufacturer  of  Sub¬ 
system  C  was  a  subcontractor.  A  prime  contractor  could  be 
more  strongly  motivated  to  aggressively  identify  corrective 
actions  and  improve  his  hardware  than  would  a  subcontractor. 
Consider  the  relative  values  of  the  ratio  of  Type  B  failures 
to  total  failures  for  the  two  subsystems: 
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C:  K2  =  27/(27  +  69)  =  28% 
G:  K2  =  18/(18  +  27)  =  40% 


Thus,  the  contractor  for  Subsystem  G  identified  a  higher  pro¬ 
portion  of  failures  as  being  correctable  than  did  the  contrac¬ 
tor  for  Subsystem  C.  This  could  explain  the  higher  value  for 
the  parameter  K2 ,  as  well  as  the  higher  value  for  K-^  relative 
to  the  predicted  MTBF. 


While  further  analyses  over  a  more  extensive  data 
base  are  clearly  needed  to  validate  the  IBM  model  and  provide 
fully  supported  methods  to  estimate  its  parameters,  the  fol¬ 
lowing  relationships  are  suggested  for  the  interim  based  on  a 
combination  of  engineering  judgement  and  the  data  analyses 
performed: 


where 


where 


=  XP  x  fa 


A  =  MIL-HDBK-217  predicted  failure  rate 


(4.3-2) 


Fa  = 


Test  acceleration  factor,  based  on  an 
assessment  of  the  degree  to  which  the 
test  environment  accelerates  the  mission 
environment  on  which  the  prediction  is 
based 


K,  =  30,000  x  F  x  a 
1  m  p 


(4.3-3) 


m 


Maturity  factor,  based  on  an  estimate  of 
failure  modes  removed  by  earlier  testing 
or  operation  and  the  imposition  of  other 
reliability  program  elements 


K2  =  (0.0005/6.5)  x  F 


A 


(4.3-4) 


4-30 


* 

,\ 


k 

k 

A 

k 

k 

i 

r~X 


Y 


r\ 

„v 


A 

& 

IR 

* 


o 
'Sj 
*%  ' 


W V\  V  y s  >  v  • 

Y,  Vt.  Cm  -  ~  S  / I  I—  l*  r* — r*  -  ■  "  1  «*— 


Using  these  suggested  relationships,  the  IBM  model  can  be  ap¬ 
plied  to  predict  reliability  growth  during  a  reliability  im¬ 
provement  development  program  as  described  in  Chapter  5. 
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5. 


RELIABILITY  GROWTH  PREDICTION  PROCEDURE 


5 . 1  OVERVIEW 

A  procedure  for  predicting  the  reliability  growth 
that  can  be  expected  for  an  equipment  undergoing  a  reliability 
growth  development  program  has  been  developed  based  on  the 
data  analyses  described  in  Chapter  4.  The  recommended  pro¬ 
cedure  is  a  departure  from  conventional  reliability  growth 
planning  in  that  the  recommended  metric  for  reliability  growth 
is  based  on  the  number  of  correctable  failure  modes  surfaced 
during  the  test  program,  rather  than  the  cumulative  MTBF  meas¬ 
ured  during  the  test.  The  IBM  model  provided  the  mathematical 
framework  for  characterizing  reliability  growth  in  this  manner. 
The  key  equipment  characteristics  and  program  attributes  in¬ 
fluencing  the  degree  of  reliability  growth  to  be  expected 
include : 

•  Equipment  complexity  as  measured  by  the 
MIL-HDBK-217  predicted  MTBF 

«  Equipment  maturity  in  terms  of  the  number 
of  failure  modes  removed  through  prior 
usage  and/or  the  imposition  of  other 
reliability  program  elements 

•  Length  of  the  test  program  in  terms  of 
the  number  of  equipment  operation  hours 
to  be  accumulated 

•  Test  environmental  conditions  relative 
to  the  operational  environment. 

These  factors  are  the  basis  for  estimating  the  initial  relia¬ 
bility  (prior  to  growth  testing)  and  predicting  the  expected 


reliability  at  the  conclusion  of  the  reliability  growth  devel¬ 
opment  test  program  using  the  procedures  described  in  the  sub¬ 
sequent  sections. 


5.2  INITIAL  RELIABILITY 

The  first  step  of  the  procedure  is  to  estimate  the 
initial  reliability.  The  initial  reliability  is  defined  to  be 
that  which  would  be  expected  for  the  equipment  if  it  were  pro¬ 
duced  and  deployed  operationally  without  being  subjected  to 
reliability  growth  testing.  It  is  assumed  that  there  are  both 
Type  A  (random)  failures  and  a  fixed  number  of  Type  B  failures 
in  the  equipment  at  the  start  of  the  test.  The  Type  A  failures 
are  characterized  by  a  constant  failure  rate  which  is  estimated 
as  the  MIL-HDBK-217  prediction,  A.  .  The  number  of.  Type  B  fail¬ 
ures  (i.e.,  the  parameter  K-^  of  the  IBM  model)  is  estimated  as 

Kj  =  30,000  x  F  x  \  (5.2-1) 

1  m  p 


where 

Fm  =  Maturity  factor,  estimated  as  the  percentage 
of  Type  B  failures  already  removed  from  the 
equipment . 

Experience  and  engineering  judgement  must  be  used  to  estimate 
Fm.  As  an  example,  if  the  equipment  is  a  modification  of  an 
existing  mature  equipment  for  a  new  application,  then  the  ma¬ 
turity  factor  should  reflect  the  percentage  of  the  design  which 
is  new. 


The  equipment  reliability,  as  measured  by  MTBF,  prior 
to  testing  can  then  be  estimated,  based  on  Eq.  2.3-5  with  t=0, 
as 
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MTBF(o)  =  [\  +  K,  x  (0,0005/6.5)] 

r  ^ 


(5.2-2) 


-1 

The  constant  (0.0005/6.5)  was  derived  in  the  data  analyses 
presented  in  Section  A.  3. 2.. 

5.3  RELIABILITY  GROWTH  DURING  TESTING 

The  next  step  is  to  predict  the  reliability  improve¬ 
ment,  relative  to  the  initial  reliability,  to  be  expected 
through  the  conduct  of  a  reliability  growth  test  of  specific 
length,  as  measured  by  equipment  operation  hours  during  test, 
t,  and  the  defined  test  environment  conditions.  Reliability 
improvement  is  measured  by  the  number  of  Type  B  failures, 
Ng(t),  identified  in  the  test.  This  can  be  estimated  based  on 
Eq.  2.3-4  as 

-  K  t 

NB(t)  =  Kx  x  (1-e  2  ) 

K-^  is  given  by  Eq.  5.2-1,  and  K2  by 

K2  =  (0.0005/6.5)  x  Fa 

where 

=  Test  acceleration  factor. 

The  numerical  value  for  the  test  acceleration  factor  should  be 
based  on  an  assessment  of  the  degree  to  which  the  rest  envi¬ 
ronment  cycle  represents  an  acceleration  of  the  operational 
environmental  cycle  which  is  the  basis  for  the  MII-HDBK-217 
prediction.  If  the  test  is  a  Combined  Environments  Reliabil¬ 
ity  Test  (CERT)  in  which  the  test  profile  is  a  simulation  of 
the  operational  profile,  the  test  acceleration  factor  is  unity 


(5.3-1) 


(5.3-2) 


If  the  test  profile  is  designed  to  reduce  periods  of  operation 
in  benign  environments,  then  the  acceleration  factor  can  be 
estimated  as 


F  =  T  /T 
A  op7  test 


(5.3-3) 


where 

T  =  Length  of  the  operational  duty  cycle 
p  (i.e.,  missing  length) 

Tt  st-  =  Length  of  the  test  cycle. 

Assuming  that  effective  corrective  actions  are  incor¬ 
porated  for  all  Type  B  failures  surfaced  during  the  test  (an 
optimistic  assumption;  see  Section  5.5),  the  equipment  MTBF 
after  performance  of  the  test  is  predicted  as 

-K0t 

MTBF(t)  =  Fa/(Fa  x  \  +  K1K2  e  1  )  (5.3-4) 

Equations  5.2-1  and  5.3-4,  in  conjunction  with  the  techniques 
described  herein  for  estimating  the  various  parameters,  pro¬ 
vide  a  methodology  for  predicting  reliability  improvement  of 
the  equipment  as  a  consequence  of  the  growth  test. 

5 . 4  EXAMPLE 

To  illustrate  application  of  the  reliability  growth 
prediction  procedure  described  above,  consider  the  following 
hypothetical  example  of  an  avionics  equipment  to  be  subjected 
to  reliability  growth  testing  during  full-scale  development. 
The  following  assumptions  are  made: 


•  40%  of  the  equipment  is  new  design;  the 
remainder  is  comprised  of  mature,  off- 
the-shelf  items 

•  The  HIL-HDBK-217  MTBF  prediction  is 

300  hours  (i.e.,  \  =  1/300) 

©  A  Test-Analvze-Fix  (TAF)  program  is  to 
be  conducted  during  which  3000  hours 
will  be  accumulated  on  the  equipment 

•  The  operational  cycle  for  the  equipment 
is  a  ten  hour  aircraft  mission 

•  The  TAF  test  profile  eliminates  the  period 
of  operation  in  a  relatively  benign  envi¬ 
ronment  (e.g.,  the  cruise  portion  of  the 
mission)  resulting  in  a  test  cycle  of 
two  hours . 


The  predicted  number  of  Type  B  failures  in  the  equipment  prior 
to  testing  (Eq.  5.2-1)  is 


K1  =  30,000  x  (0.4)  x  (1/300) 
=  40 


The  initial  MTBF  (Eq.  5.2-2)  is 

MTBF(o)  =  [1/300  +  40  x  (0 . 0005/6 . 5 )] "1 
=  156  hours 


The  test  acceleration  factor  (Eq.  3.3-3)  is 


Fa  =  10/2 
=  5 

The  rate  of  surfacing  Type  B  failures  during  the  TAF  test 
(Eq.  5.3-2)  is 


K2  =  ( 0 . 00C5/6 . 5 )  X  5 


=  0 . 0C03846 

The  number  of  Type  B  failures  identified  during  the  test 
(Eq.  5.3-1)  is 

Ng(30Q0)  =  40  x  (1  -  e"0 • 0003846  x  3000^ 

=  27 

The  equipment  MTBF  after  incorporation  of  corrective  actions 
to  eliminate  those  Type  B  failures . identified  in  the  TAF  pro¬ 
gram  (Eq.  5.3-4)  is 

MTBF(3C00)  =  5/(5  x  1/300  +  40  x  0.0003846  e‘°* 0003846  X  300°) 

=  232  hours 

Hence,  the  predicted  reliability  growth  is  from  an  initial 
MTBF  of  156  hours  to  an  improved  MTBF  of  232  hours,  approxi¬ 
mately  a  50%  improvement. 


5.5  FOLLOW-ON  TEST  PHASES 

The  reliability  growth  prediction  procedure  described 
above  applies  to  a  single  phase  of  reliability  growth  testing. 
If  the  reliability  growth  development  program  is  co  be  con¬ 
ducted  in  multiple  phases  (see  Section  2.2.1),  then  the  pro¬ 
cedure  can  be  applied  sequentially  to  predict  the  expected 
incremental  reliability  growth  in  each  test  phase.  The  prin¬ 
cipal  consideration  is  that  the  assumed  number  of  Type  B  fail¬ 
ures  at  the  start  of  a  follow-on  test  phase  should  reflect  che 
elimination  of  these  Type  B  failures  identified  in  prior  test 
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phases.  For  example,  if  ^  denotes  the  equivalent  of  the 
parameter  K-^  for  a  second  phase  of  testing,  then 

K1 ,2  =  Kx  -  Nfi(t)  (5.5-1) 

where  and  Ng(t)  are  given  by  Eq.  5.2-1  and  Eq.  5.3-1,  re¬ 
spectively.  Additionally,  the  distinct  characteristics  of  the 
test  program  in  each  phase  should  be  reflected  in  the  test- 
related  parameters.  Specifically,  the  test  length,  t,  and 
test  acceleation  factor,  E'^,  will  in  all  likelihood  be  unique 
for  each  phase. 

If  the  testing  is  conducted  in  multiple  phases,  an 
opportunity  will  be  provided  to  assess  the  effectiveness  of 
corrective  actions  incorporated  to  eliminate  Type  B  failures 
identified  in  earlier  test  phases.  The  prediction  procedure 
as  described  yields  an  upper  bound  on  reliability  growth  in 
that  100%  fix-effectiveness  of  Type  B  failures  is  assumed.  A 
fix-effectiveness  factor  can  be  derived  as  follows: 

fE  =  1  -  nb,R/NB  (5.5-2) 

where 

N~  =  Number  of  Type  B  failure  identified 
in  prior  test  phases  with  corrective 
actions  incorporated 

Ng  r  =  Number  of  Type  B  failures  which  have 
’  recurred  after  incorporation  of 
corrective  action 

l 

The  fix-effectiveness  factor  thus  provides  a  means  for  scaling  c 

the  upper  bound  reliability  growth  prediction  to  account  for 
imperfect  corrective  actions. 
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6. 


RELIABILITY  GROWTH  GUIDELINES 


6.1  SELECTION  OF  EQUIPMENT  DEVELOPMENT  PROGRAMS 

It  is  easily  argued  that  reliability  improvement  po¬ 
tential  exists  in  all  systems,  that  any  reliability  improve¬ 
ment  is  good  and,  hence,  reliability  growth  testing  should  be 
imposed  on  all  equipment  development  programs.  However,  one 
of  the  findings  of  this  study  was  that  reliability  growth  test 
ing  is  not  cost-effective  for  some  equipments.  Therefore,  a 
rational  procedure  is  needed  to  determine  whether  to  impose 
such  testing.  A  four- step  procedure  is  recommended: 

1.  Determine  feasibility  of  growth  testing 
and  associated  constraints 

2.  Assess  current  reliability  and  the  poten¬ 
tial  for  improvement 

3.  Determine  need  rcr  improvement  to  meet 
user  requirements 

4.  Estimate  cost-effectiveness. 

Each  step  is  described  below. 

6.1.1  Feasibility  and  Constraints 

Availability  of  Facilities  -  Meaningful  reliability 
growth  testing  of  avionics  equipment  can  only  be  conducted 
when  the  system  can  be  subjected  to  combined  environmental 
conditions  (temperature,  vibration,  humid-'  ty)  at  extremes  rep¬ 
resentative  of  the  mission  environment.  This  implies  the  re¬ 
quirement  for  a  specialized  test  chamber,  preferably  at  the 


manufacturer  facility  or,  alternatively,  at  an  independent 
test  laboratory.  The  chamber  must  be  large  enough  for  all 
equipments  to  be  tested.  Whether  all  equipments  need  to  be 
tested  together  as  an  integrated  system,  in  order  to  verify 
system  operation  and  exercise  interfaces,  is  a  consideration. 
The  need  for  software  to  simulate  external  interfaces  must 
also  be  addressed. 

Schedule  Constraints  -  Reliability  growth  testing 
requires  time,  generally  at  least  six  months.  The  urgency  of 
the  user's  need,  and  the  attendant  program  schedule,  must  be 
evaluated  to  determine  if  the  time  is  available.  If  the  sched¬ 
ule  is  severely  constrained,  then  the  objectives  of  the  growth 
test  could  be  defeated,  and  it  may  be  more  beneficial  to  im¬ 
prove  reliability  through  che  imposition  of  other  reliability 
program  elements  (e.g. ,  high  reliability  parts)  during  the 
test  program. 

Availability  of  Test  Articles  -  System  prototypes 
must  be  dedicated  tc  the  growth  test.  If  provisions  for  such 
prototypes  were  not  incluaed  initially,  then  the  program  must 
be  reviewed  to  determine  if  conflicting  demands  for  test  arti¬ 
cles  (e.g.,  flight  testing)  preclude  such  dedication.  The 
test  article  should  be  complete.  While  some  growth  tests  have 
been  conducted  on  partial  systems,  this  can  limit  the  ability 
to  verify  operational  performance.  Further,  it  is  often  the 
missing  segments  still  undergoing  development  which  are  the 
"tail-poles"  relative  to  reliability. 

Cost  Constraint  -  Obviously,  if  reliability  growth 
testing  is  imposed  by  the  program  office,  the  FSD  budget  must 
include  the  associated  cost.  This  includes  the  cost  of  the 
test  articles,  chamber  operations,  and  engineering  labor  to 
analyze  test  failures  and  identify  corrective  actions.  If 


only  a  limited  amount  of  funds  can  be  targeted  for  growth  test¬ 
ing,  it  must  be  determined  if  a  meaningful  growth  test,  e.g., 
of  sufficient  length,  can  be  supported.  The  risk  of  stretch¬ 
ing  available  funds  too  thin  is  a  lack  of  aggressiveness  by 
the  contractor  to  improve  the  equipment. 


6.1.2  Reliability  Improvement  Potential  Assessment 


A  lesson- learned  from  the  B-52  OAS  TAF  is  that  there  is 
minimal  benefit  derived  from  imposing  reliability  growth  test¬ 
ing  on  mature  equipments.  An  assessment  should  be  performed 
of  both  the  current  expected  operational  reliability  of  the 
system,  i.e.,  if  growth  testing  is  not  imposed,  and  the  degree 
of  improvement  that  can  be  achieved  through  reliability  growth 
testing  within  the  constraints  discussed  in  Section  6.1.1, 


Current  Reliability  -  Clearly,  the  best  means  for 
assessing  current  reliability  is  operational  data,  if  they  exist. 
In  the  absence  of  such  data,  or  equivalent  data  on  analogous 
systems  manufactured  by  the  contractor,  the  method  described 
in  Section  5.2  can  be  applied.  This  approach  is  recommended 
rather  than  the  Duane  10%  rule  because  of  the  ambiguities  in 
the  Duane  initial  conditions.  It  focuses  attention  on  the 
component  of  reliability  which  can  be  improved  through  correc¬ 
tive  actions,  i.e.,  Type  B  failures,  rather  than  on  burn-in 
type  failures  which  would  be  more  effectively  removed  by  Envi¬ 
ronmental  Stress  Screening  in  production. 


Growth  Potential  -  Section  5.3  provides  procedures 
for  predicting  the  operational  reliability  growth  achievable 
in  a  test  of  specified  environmental  conditions  and  length. 
Using  these  procedures,  the  operational  reliability  at  the 
conclusion  of  the  test  can  be  predicted.  The  current  and  final 
reliability  predictions,  as  estimated  using  Eqs .  5.2-2  and 
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5.3-4,  respectively,  provide  the  basis  for  assessing  the  need 
for  and  cost-effectiveness  of  reliability  growth  testing  as 
discussed  next. 

6.1.3  Need  Assessment 

Assessment  of  the  need  for  reliability  growth  is  simple 
and  straightforward.  The  current  reliability,  MTBF(o),  is 
compared  to  the  user  requirement  as  stated  in  the  Statement  of 
Need  (SON),  Required  Operational  Capability  (ROC),  Mission 
Element  Need  Statement  (MENS),  etc.  If  the  user  requirement 
is  not  achieved  by  the  current  reliability,  then  reliability 
improvement  is  clearly  necessary.  If  the  requirement  is  met, 
then  the  imposition  of  growth  testing  should  be  based  on  a 
cost-effectiveness  analysis. 
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6.1.4  Cost-Effectiveness 

Reliability  improvement  results  in  reduced  initial 
logistics  support  (e.g.,  spares  to  meet  system  readiness  re¬ 
quirements)  and  operations  &  support  (e.g.,  maintenance  labor 
and  materials)  costs.  Generally,  the  greater  the  quantity  of 
systems  to  be  deployed  and  the  higher  the  utilization,  the 
greater  the  savings  downstream.  Both  the  costs  of  reliability 
improvement  and  the  resulting  cost  reduction  can  be  quantified 
The  improvement  cost  includes  fixed  costs  (e.g.,  test  article 
fabrication)  and  costs  which  are  a  function  of  test  length 
(e.g.,  chamber  operations).  A  life-cycle  cost  model,  Ref.  14 
for  example,  can  be  exercised  to  quantify  the  cost  savings 
associated  with  improving  reliability  from  MTBF(o)  to  MTBF(t). 
Hence,  the  cost  of  conducting  a  test  of  length  t  to  improve 
reliability  from  MTBF(o)  to  MTBF(t)  can  be  traded  off  against 
the  downstream  savings,  i.e.,  LCC  (at  MTBF(o))  minus  LCC  (at 
MTBF(t)),  to  determine  cost  effectiveness. 
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6.2 


SPECIFICATION  GUIDELINES 


The  principal  recommended  modifications  to  current 
reliability  growth  testing  and  specification  practices  pertain 
to  an  increased  emphasis  on  failure  classification  (Type  A  vs 
Type  B)  and  corrective  action  tracking.  Also,  a  follow-on 
test  to  permit  incorporation  of  all  corrective  actions  into 
test  articles  is  recommended  subsequent  to  the  primary  TAF 
test,  the  purpose  being  to  verify  the  effectiveness  of  cor¬ 
rective  actions  and  measure  operational  reliability. 

6.2.1  Planned  Growth  Curves 

MIL-HDBK  189  discusses  the  reliability  growth  that 
occurs  over  an  entire  development  program.  The  overall  pat¬ 
tern  is  described  by  an  idealized  growth  curve,  as  shown  by 
the  dashed  line  in  Fig.  6.2-1.  The  solid  lines  are  the  plan¬ 
ning  curves  that  are  relevant  to  particular  test  phases  within 
the  program  itself.  This  report  deals  with  predicting  the 
growth  that  should  be  observed  during  a  Test-Auialyze-Fix  (TAF) 
test  phase. 

When  tracking  reliability  during  a  major  test  phase 
within  a  development  program,  such  as  a  TAF,  there  are  three 
reliability  values  that  are  of  interest  on  the  planned  growth 
curve .  They  are : 

•  Reliability  predicted  at  the  beginning 

of  the  test  phase 

•  Reliability  achieved  by  incorporating 

fixes  during  the  test 

t-  Reliability  achieved  by  introducing  de¬ 
layed  fixes  into  the  system  at  the  end 

of  the  test. 


Figure  6.2-1  Development  Program  Idealized 

and  Planned  Growth 

The  situation  is  depicted  pictorially  in  Fig.  6.2-2.  A  planned 
growth  curve  taking  these  considerations  into  account  should 
be  established  prior  to  start  of  reliability  growth  testing. 

6.2.2  Test  Environment 

The  test  environment  should  be  designed  to  surface 
equipment  design  or  manufacturing  problems  which  can  be  cor¬ 
rected  in  order  to  improve  reliability.  There  are  two  forms 
of  test  acceleration: 

•  Time  Acceleration 

6 


Environmental  Acceleration. 
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MILESTONE  AFTER 
DELAYED  FIXES 


MILESTONE  AT 
BEGINNING  OF 
TEST  PHASE 


MILESTONE  AT 
END  OF 
TEST  PHASE 


Figure  6.2-2 


TEST  PHASE 


Reliability  Values  Associated 
With  A  Major  Test  Phase 


In  a  time -accelerated  test,  the  simulated  environmental  pro¬ 
file  deemphasizes  the  benign  mission  phases  that  are  assumed 
not  to  generate  many  failures.  The  benign  phases  include  ex¬ 
tended  cruise  and  ground  time.  Accordingly,  most  of  the  test 
cycle  is  devoted  to  the  more  stringent  environments,  which  for 
avionics  are  associated  with  takeoff,  climbout. ,  and  altitude 
changes . 


In  an  environment ally -accelerated  test,  the  environ¬ 
mental  levels  employed  are  more  severe  than  those  associated 
with  a  typical  mission  profile  (or  with  the  design  limits  of 
the  equipment).  Accelerated  environments  are  used  to  detect 
equipment  failure  modes  that  would  be  exhibited  only  under 
exti*eme  flight  conditions,  or  would  not  be  observed  until  the 
equipment  is  exposed  to  normal  conditions  for  the  greater 
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periods  of  time  associated  with  equipment  wear-out  modes. 
Environmental  acceleration  imposes  the  risk  of  inducing  non- 
relevant  failure  modes. 


The  time  and  environmental  acceleration  conditions 
should  then  he  translated  into  a  test  environmental  profile  as 
a  basis  for  specifying  test  chamber  operations.  Figure  6.2-3 
is  an  example  of  such  a  profile. 

6.2.3  Length  of  Test 

The  length  of  testing  should  be  chosen  so  as  to  gener¬ 
ate  sufficient  experience  with  specific  failure  modes  so  that 
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Figure  6.2-3 


Representative  Test  Environmental  Profile 
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6.2.4  Data  Requirements 
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corrective  action  may  be  identified  and  implemented.  It  is 
very  important  that  the  test  hours  are  either  long  enough  to 
allow  corrective  actions  to  be  incorporated  or  flexible  enough 
to  permit  lengthening  the  scheduled  test  time.  The  contrac¬ 
tors  should  be  encouraged  to  incorporate  corrective  actions 
and  to  verify  them  in  TAF ;  therefore,  they  should  be  allowed 
control  over  equipment  configurations  while  being  required  to 
provide  an  audit  trail  of  configurations  to  the  test  monitor. 
The  procedure  described  in  Section  6.1.4  can  be  utilized  to 
specify  a  test  length  that  is  cost-effective. 


& 


,V 

I 

p 


v 

,v 


-  U 

L  ,  * 


Data  requirements  during  test  must  be  specif iea. 
Procedures  must  be  written  such  that  no  subjective  determina¬ 
tions  are  requited  on  the  part  of  the  test  operators.  Failure 
criteria  .should  be  consistent  with  those  applied  in  the  antici¬ 
pated  operational  environment.  Detailed  procedures  should  be 
specified  for  recording  equipment  "on  time”  with  each  test 
event  noted  in  the  data  logs.  If  elapsed  time  indicator  (ETI) 
meters  are  installed  in  the  equipment,  these  should  be  used  as 
the  "on  time"  baseline;  ETI  values  should  be  recorded,  as  a 
minimum,  at  the  beginning  and  end  of  each  test  cycle/sortie. 
Serial  numbers  of  each  LRU  undergoing  test  should  be  recorded 
prior  to  each  test  cycle  for  the  purpose  of  correlating  sub¬ 
sequent  equipment  performance  with  configuration. 

With  the  above  requirements  in  mind,  the  key  TAF  data 
elements  are  identified  in  Table  6.2-1.  This  list  is  consid¬ 
ered  to  be  the  minimum  level  of  data  required  to  track  and 
assess  reliability  growth. 

With  regards  to  reporting  of  the  above  data  elements, 
it  is  necessary  to  specify  reporting  requirements  with  respect 
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to  time.  However,  the  reporting  structure  should  not  be  so 
stringent  as  to  prevent  the  contractor  from  performing  de¬ 
tailed  failure  analysis  in  order  to  meet  a  reporting  require¬ 
ment.  It  should  be  possible,  within  the  framework  of  the  re¬ 
porting  requirements,  for  specific  failure  report  to  be  allowed 
to  remain  open  pending  the  outcome  of  analysis.  For  example, 
if  failure  analysis  is  incomplete,  the  contractor  should  sub¬ 
mit  the  required  documentation  presenting  what  has  been  accom¬ 
plished  up  to  that  point  and  stating  that  failure  analysis  is 
on-going.  At  some  other  point  in  time,  e.g.,  30  days  later, 
the  failure  analysis  should  be  readdressed  and  either  closed 
out  or  continued  with  further  updates  of  on-going  analysis. 


6.2.5  Configuration  Tracking 


A  separate  log  should  be  maintained  of  all  corrective 
actions  identified  as  the  result  of  the  failure  analyses. 
This  should  include  for  each  corrective  action: 


•  The  elapsed  time  at  which  each  failure 
associated  with  the  corrective  action 
occurred  and  the  serial  number  of  the 
test  item. 

•  The  elapsed  time  at  which  the  corrective 
action  was  incorporated  into  each  unit 
under  test  by  serial  number. 


These  data  will  allow  estimation  of  reliability  improvement  in 
accordance  with  the  procedure  described  in  Chapter  5,  as  well 
as  verification  of  the  effectiveness  of  each  corrective  action. 

6.2.6  Follow-Up  Testing 

A  follow-up  to  the  reliability  growth  test  is  recom¬ 
mended  allowing  sufficient  time  for  incorporation  of  all  correc 
tive  actions.  The  follow-up  test  should  be  a  verification,  as 
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opposed  to  a  growth,  test.  As  such,  the  environmental  profile 
should  be  representative  of  the  mission  profile,  equipment 
configuration  should  be  controlled  and  the  test  should  be  of 
sufficient  length  to  allow  demonstration  of  the  target  MTBF 
with  sta  istical  confidence.  MIL-STD-781  provides  guidelines 
for  structuring  such  a  test. 
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7.  CONCLUSIONS 


7.1  RELIABILITY  GROWTH  TESTING:  PRACTICES  AND  PITFALLS 

Several  findings  emerged  from  the  study  relating  to 
reliability  growth  testing  as  it  has  been  performed  in  the 
past  and  the  true  meaning  of  reliability  growth  curves  as  they 
are  conventionally  presented. 
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7.1.1  Reliability  Growth  Testing  is  Conducted  in  A 
Time-Constrained  Environment 
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The  fpcus  of  this  study  was  on  reliability  growth 
testing  during  full-scale  development,  as  opposed  to  the  pro¬ 
duction  and  operational  phases  of  the  system  life  cycle.  In 
each  of  the  development  programs  surveyed  in  this  study,  the 
amount  of  calendar  time  available  to  perform  growth  testing 
was  limited  to  less  than  a  year,  usually  only  six  months. 
While  it  could  be  argued  that  the  available  test  time  could  be 
expanded  by' starting  testing  earlier,  the  test  articles  should 
be  reasonably  representative  of  production  prototypes.  Simi¬ 
larly,  there  are  constraints  on  the  number  of  prototypes  which 
can  be  dedicated  to  reliability  growth  testing  due  to  funding 
limitations  and  other  test  agency  demands.  Consequently,  the 
programs  surveyed  generally  entailed  intensive  testing  of  a 
small  number  of  prototypes  (usually  two)  continuously  over  a 
limited  interval  of  time. 
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7.1.2  Reliability  Growth  as  Conventionally  Portrayed 


Can  be  Misleading 


Clearly,  true  reliability  improvement  can  only  result 
from  the  incorporation  of  design  changes  and  other  corrective 
actions.  However,  the  test  scenario  described  above  does  not 
allow  timely  incorporation  of  corrective  actions  into  the  units 


undergoing  test,  the  reason  being  that  when  a  failure  occurs, 
the  test  unit  is  repaired  and  entered  back  into  test  as  soon 
as  possible.  The  process  of  failure  analysis  and  corrective 
action  identification  is  conducted  in  parallel.  At  best,  cor¬ 
rective  actions  are  not  incorporated  into  the  test  articles 
until  the  tail-end  of  the  test  and,  more  often,  are  not  incor¬ 
porated  until  production.  Hence,  the  results  of  corrective 
actions  are  not  represented  in  the  failure/time  history  data 
generated  in  the  growth  test.  Still  the  data  do  indicate  reli¬ 
ability  growth.  This  "apparent''  growth  is  best  explained  by 
Swett  (Ref.  15)  as  follows: 


"The  weakest  part  of  the  test  article  fails  first. 

The  replacement  part  is  a  random  sample  from  the 
spare  parts  inventory.  That  means  there  may  be  ten- 
thousand-  to-one  odds  that  the  replacement  is  a  better 
part  than  the  one  it  replaces.  So  the  composite 
reliability  of  parts  in  the  test  article  is  being 
systematically  improved  each  time  there  is  a  failure, 
and  observed  reliability  of  the  test  article  continues 
to  grow. " 


It  was  also  demonstrated  in  this  study  that  the  prac¬ 
tice  of  plot* mg  cumulative  data,  combined  with  infant  mortal¬ 
ity  in  test  articles,  can  lead  to  the  appearance  of  growth 
when,  in  fact,  none  is  occurring.  The  implication  relative  to 
the  objectives  of  this  study  is  that  it  is  not  sufficient  to 
draw  conclusions  from  historical  growth  curves  alone;  it  is 
necessary,  instead,  to  factor  into  consideration  the  number 
and  nature  of  the  corrective  actions  generated  as  a  conse¬ 
quence  cf  the  test. 
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7.2 


SYSTEM  COMPLEXITY,  MATURITY  AND  GROWTH  TEST  CONDITIONS 
.ARE  THE  FACTORS  MOST  STRONGLY  RELATED  TO  RELIABILITY 
GROWTH 


Reliability  growth,  as  measured  by  corrective  actions 
identified  or  by  failure  modes  eliminated  through  corrective 
action,  was  found  to  be  related  to  the  complexity  of  the  equip¬ 
ment  as  reflected  by  predicted  MTBF .  The  MIL-HDBK-217  MTBF 
prediction  is  a  meaningful  measure  of  complexity,  representing 
as  it  does  both  the  total  number  of  parts  in  the  equipment  and 
the  complexity  cf  the  parts  themselves.  Additional  analyses 
demonstrated  that  the  highest  level  of  corrective  action  activ¬ 
ity  was  generated  for  the  least  mature  equipments  included  in 
the  program  survey  and  the  the  least  corrective  actions  were 
observed  on  the  most  mature  equipment  item.  Thus,  maturity  as 
characterized  by  the  degree  of  new  hardware  development,  is 
related  to  reliability  growth. 

The  environmental  stress  levels  and  time-compression 
factors  associated  with  the  reliability  growth  test  profile 
were  also  found  to  influence  reliability  growth.  A  higher 
degree  of  growth  was  observed  in  these  programs  for  which  the 
test  profile  was  designed  to  stimulate  failures,  as  onposed  to 
simulating  mission  conditions. 


7.3  RELIABILITY  GROWTH  PREDICTION 

7.3.1  Historically-Observed  Relationships  were  Confirmed 

Analysis  of  the  failure- time  history  data  generated 
in  the  programs  surveyed  confirmed  that: 

•  Such  data  do  tend  to  track  to  a  Duane 
model 


•  The  Duane  reliability  growth  rate  (a)  is 
within  the  historically  observed  bounds, 
0.1  <  a  <  0.7 

•  While  there  is  considerable  variation 
for  specific  items,  the  Duane  initiali¬ 
zation  groundrules  given  in  MIL-STD-1635 
are,  on  the  average,  reasonable  if  no 
other  information  is  available. 


The  growth  rate,  a,  was  found  to  be  strongly  influenced  by  the 
occurrence  of  failures  early  in  the  test,  an  influence  which 
tended  to  mask  out  any  underlying  relationships  between  growth 
rate  and  equipment  characteristics  or  program  attributes. 
However,  the  following  quantifiable  relationships  to  test  con¬ 
ditions  were  identified: 


•  a  =  0.5,  for  a  TAF-type  test  designed  to 
stimulate  equipment  failures 

•  a  =  0.35,  for  a  test  designed  to  simu¬ 
late  the  mission  environment 

•  a  =  0.2,  for  a  benign  operational  test. 

These  relationships  should  be  accepted  in  the  mean-value  sense 
only,  recognizing  that  there  will  be  variation  about  the  mean 
for  any  specific  program  based  on  the  aggressiveness  of  the 
contractor's  corrective  action  program 

7.3.2  Alternatives  to  the  Duane  and  AMSAA  Models 

are  Recommended  for  Reliability  Growth  Prediction 

One  of  the  major  conclusions  of  this  study  is  that 
the  two  models  (Duane  and  AMSAA)  which  are  widely  used  today 
for  reliability  growth  measurement  and  management  have  limita¬ 
tions  as  predictive  tools.  The  limitations  are  two- fold: 


The  models  do  not  discriminate  between 
the  apparent  reliability  growth  due  to 
extended  bum-in  effects  and  true  relia¬ 
bility  growth  associated  with  the  identi¬ 
fication  and  incorporation  of  design 
changes  and  other  corrective  actions 

The  parameters  of  the  model  are  empirical 
in  nature  (i.e.,  determined  by  data  fits) 
and  cannot  be  intuitively  related  to 
equipment  characteristics  and  program 
attributes . 


It  was  deemed  desirable  that  prediction  techniques  instead  be 
based  on  a  model  whose  parameters  are  more  directly  related  to 
equipment  characteristics  and  program • attributes ,  and  which 
addresses  corrective  action  potential.  A  promising  candidate, 
the  IBM  model,  was  evaluated. 

A  procedure  for  applying  the  IBM  model  to  reliability 
growth  data  was  identified,  providing  that  the  data  include 
failure  classifications  (random  vs.  correctable).  The  model 
was  applied  to  two  data  sets  in  the  data  base  and  found  to 
yield  reasonable  fits.  Intuitively  plausible  relationships 
were  developed  for  estimating  the  parameters  of  the  model  based 
on: 

•  MIL-HDBK-217  predicted  reliability 

•  Equipment  maturity 

m  Time  acceleration  factor  associated  with 

the  growth  test  profile 
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Test  length, 


While  further  study  will  be  required  to  validate  the  model  and 
refine  the  constants  developed  for  these  relationships,  the 
model  provides  an  interim  capability  for  reliability  growth 
prediction. 
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7.4 


RECOMMENDATIONS 


The  Reliability  Growth  Prediction  Study  has  surfaced 
the  need  to  focus  less  attention  on  cumulative  failure- time 
data  plotted  on  a  log-log  scale  and  more  on  failure  model 
classification  and  corrective  action  identification  and  veri¬ 
fication.  Toward  that  need,  the  following  specific  recommen¬ 
dations  are  offered: 


•  Institutionalize  procedures  for  failure 
classification  and  corrective  action 
tracking  in  future  reliability  develop¬ 
ment  growth  testing  procedures 

•  Require  performance  of  a  reliability 
demonstration  test,  subsequent  to  the 
growth  test,  for  verification  of  correc¬ 
tive  action  effectiveness 

•  Develop  procedures  for  centralizing  the 
data  generated  during  such  tests 

•  As  data  accumulated,  perform  further 

research  to  validate  the  reliability 
growth  prediction  procedure  suggested 
herein. 
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of 

Rome  Air  Development  Center 

ZAVC  plant)  and  executes  AeseaAch,  development,  test  and 
selected  acquisition  pn.oqn.am  in  suppoAt  o $  Command,  Control 
Communications  and  Intelligence  (C^X)  activiti.es.  Technical 
and  engineering  support  within  ajieas  o&  technical  competence 
is  provided  to  BSV  ?n.oqn.am  O^ice s  (PcXs)  and  other  BSD 
elements,  The  principal  technical  mission  areas  one 
communications ,  ele.ctromagnetic  guidance  and  control,  sua- 
veillance  ofi  ground  and  aeAospace  objects,  intelligence  data 
1 ollzotion  and  handling ,  information  system  technology, 
ionospheric  pAopagation,  solid  state  sciences,  microwave 
physics  and  electAonic  Aeliability,  maintainability  and 
compatibility. 
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